]> Akamai

rmarx@akamai.com

Meta

lniccolini@meta.com

Protocol Labs

marten@protocol.ai

Cloudflare

lucaspardue.24.7@gmail.com

Transport QUIC Internet-Draft This document describes a high-level schema for a standardized logging format called qlog. This format allows easy sharing of data and the creation of reusable visualization and debugging tools. The high-level schema in this document is intended to be protocol-agnostic. Separate documents specify how the format should be used for specific protocol data. The schema is also format-agnostic, and can be represented in for example JSON, csv or protobuf.

Introduction There is currently a lack of an easily usable, standardized endpoint logging format. Especially for the use case of debugging and evaluating modern Web protocols and their performance, it is often difficult to obtain structured logs that provide adequate information for tasks like problem root cause analysis. This document aims to provide a high-level schema and harness that describes the general layout of an easily usable, shareable, aggregatable and structured logging format. This high-level schema is protocol agnostic, with logging entries for specific protocols and use cases being defined in other documents (see for example for QUIC and for HTTP/3 and QPACK-related event definitions). The goal of this high-level schema is to provide amenities and default characteristics that each logging file should contain (or should be able to contain), such that generic and reusable toolsets can be created that can deal with logs from a variety of different protocols and use cases. As such, this document contains concepts such as versioning, metadata inclusion, log aggregation, event grouping and log file size reduction techniques. The qlog schema can be serialized in many ways (e.g., JSON, CBOR, protobuf, etc). This document describes only how to employ , its subset , and its streamable derivative .

• Note to RFC editor: Please remove the follow paragraphs in this section before publication.

Feedback and discussion are welcome at https://github.com/quicwg/qlog. Readers are advised to refer to the "editor's draft" at that URL for an up-to-date version of this document. Concrete examples of integrations of this schema in various programming languages can be found at https://github.com/quiclog/qlog/.

Notational Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Schema definition To define events and data structures, all qlog documents use the Concise Data Definition Language . This document uses the basic syntax, the specific text, uint, float32, float64, bool, and any types, as well as the .default, .size, and .regexp control operators, the ~ unwrapping operator, and the $ extension point syntax from . Additionally, this document defines the following custom types for clarity:

Additional CDDL type definitions

All timestamps and time-related values (e.g., offsets) in qlog are logged as float64 in the millisecond resolution. Other qlog documents can define their own CDDL-compatible (struct) types (e.g., separately for each Packet type that a protocol supports).

• Note to RFC editor: Please remove the following text in this section before publication.

The main general CDDL syntax conventions in this document a reader should be aware of for easy reading comprehension are:

• ? obj : this object is optional
• TypeName1 / TypeName2 : a union of these two types (object can be either type 1 OR type 2)
• obj: TypeName : this object has this concrete type
• obj: [* TypeName] : this object is an array of this type with minimum size of 0 elements
• obj: [+ TypeName] : this object is an array of this type with minimum size of 1 element
• TypeName = ... : defines a new type
• EnumName = "entry1" / "entry2" / entry3 / ...: defines an enum
• StructName = { ... } : defines a new struct type
• ; : single-line comment
• * text => any : special syntax to indicate 0 or more fields that have a string key that maps to any value. Used to indicate a generic JSON object.

All timestamps and time-related values (e.g., offsets) in qlog are logged as float64 in the millisecond resolution. Other qlog documents can define their own CDDL-compatible (struct) types (e.g., separately for each Packet type that a protocol supports).

Serialization examples Serialization examples in this document use JSON () unless otherwise indicated.

Design goals The main tenets for the qlog schema design are:

• Streamable, event-based logging
• A flexible format that can reduce log producer overhead, at the cost of increased complexity for consumers (e.g. tools)
• Extensible and pragmatic
• Aggregation and transformation friendly (e.g., the top-level element for the non-streaming format is a container for individual traces, group_ids can be used to tag events to a particular context)
• Metadata is stored together with event data

The high level qlog schema A qlog file should be able to contain several individual traces and logs from multiple vantage points that are in some way related. To that end, the top-level element in the qlog schema defines only a small set of "header" fields and an array of component traces. For this document, the required "qlog_version" field MUST have a value of "0.3".

Note:

there have been several previously broadly deployed qlog versions based on older drafts of this document (see draft-marx-qlog-main-schema). The old values for the "qlog_version" field were "draft-00", "draft-01" and "draft-02". When qlog was moved to the QUIC working group, we decided to switch to a new versioning scheme which is independent of individual draft document numbers. However, we did start from 0.3, as conceptually 0.0, 0.1 and 0.2 can map to draft-00, draft-01 and draft-02.

As qlog can be serialized in a variety of ways, the "qlog_format" field is used to indicate which serialization option was chosen. Its value MUST either be one of the options defined in this document (e.g., ) or the field must be omitted entirely, in which case it assumes the default value of "JSON". In order to make it easier to parse and identify qlog files and their serialization format, the "qlog_version" and "qlog_format" fields and their values SHOULD be in the first 256 characters/bytes of the resulting log file. An example of the qlog file's top-level structure is shown in . Definition:

QlogFile definition

JSON serialization example:

QlogFile example

Summary In a real-life deployment with a large amount of generated logs, it can be useful to sort and filter logs based on some basic summarized or aggregated data (e.g., log length, packet loss rate, log location, presence of error events, ...). The summary field (if present) SHOULD be on top of the qlog file, as this allows for the file to be processed in a streaming fashion (i.e., the implementation could just read up to and including the summary field and then only load the full logs that are deemed interesting by the user). As the summary field is highly deployment-specific, this document does not specify any default fields or their semantics. Some examples of potential entries are shown in . Definition:

Summary definition any } ]]>

JSON serialization example:

Summary example

traces It is often advantageous to group several related qlog traces together in a single file. For example, we can simultaneously perform logging on the client, on the server and on a single point on their common network path. For analysis, it is useful to aggregate these three individual traces together into a single file, so it can be uniquely stored, transferred and annotated. As such, the "traces" array contains a list of individual qlog traces. Typical qlogs will only contain a single trace in this array. These can later be combined into a single qlog file by taking the "traces" entry/entries for each qlog file individually and copying them to the "traces" array of a new, aggregated qlog file. This is typically done in a post-processing step. The "traces" array can thus contain both normal traces (for the definition of the Trace type, see ), but also "error" entries. These indicate that we tried to find/convert a file for inclusion in the aggregated qlog, but there was an error during the process. Rather than silently dropping the erroneous file, we can opt to explicitly include it in the qlog file as an entry in the "traces" array, as shown in . Definition:

TraceError definition

JSON serialization example:

TraceError example

Note that another way to combine events of different traces in a single qlog file is through the use of the "group_id" field, discussed in .

Individual Trace containers The exact conceptual definition of a Trace can be fluid. For example, a trace could contain all events for a single connection, for a single endpoint, for a single measurement interval, for a single protocol, etc. As such, a Trace container contains some metadata in addition to the logged events, see . In the normal use case however, a trace is a log of a single data flow collected at a single location or vantage point. For example, for QUIC, a single trace only contains events for a single logical QUIC connection for either the client or the server. The semantics and context of the trace can mainly be deduced from the entries in the "common_fields" list and "vantage_point" field. Definition:

Trace definition

JSON serialization example:

Trace example

Configuration We take into account that a qlog file is usually not used in isolation, but by means of various tools. Especially when aggregating various traces together or preparing traces for a demonstration, one might wish to persist certain tool-based settings inside the qlog file itself. For this, the configuration field is used. The configuration field can be viewed as a generic metadata field that tools can fill with their own fields, based on per-tool logic. It is best practice for tools to prefix each added field with their tool name to prevent collisions across tools. This document only defines two optional, standard, tool-independent configuration settings: "time_offset" and "original_uris". Definition:

Configuration definition any } ]]>

JSON serialization example:

Configuration example

time_offset The time_offset field indicates by how many milliseconds the starting time of the current trace should be offset. This is useful when comparing logs taken from various systems, where clocks might not be perfectly synchronous. Users could use manual tools or automated logic to align traces in time and the found optimal offsets can be stored in this field for future usage. The default value is 0.

original_uris The original_uris field is used when merging multiple individual qlog files or other source files (e.g., when converting .pcaps to qlog). It allows to keep better track where certain data came from. It is a simple array of strings. It is an array instead of a single string, since a single qlog trace can be made up out of an aggregation of multiple component qlog traces as well. The default value is an empty array.

custom fields Tools can add optional custom metadata to the "configuration" field to store state and make it easier to share specific data viewpoints and view configurations. Two examples from the qvis toolset are shown in .

Custom configuration fields example

vantage_point The vantage_point field describes the vantage point from which the trace originates, see . Each trace can have only a single vantage_point and thus all events in a trace MUST BE from the perspective of this vantage_point. To include events from multiple vantage_points, implementers can for example include multiple traces, split by vantage_point, in a single qlog file. Definitions:

VantagePoint definition

JSON serialization examples:

VantagePoint example

The flow field is only required if the type is "network" (for example, the trace is generated from a packet capture). It is used to disambiguate events like "packet sent" and "packet received". This is indicated explicitly because for multiple reasons (e.g., privacy) data from which the flow direction can be otherwise inferred (e.g., IP addresses) might not be present in the logs. Meaning of the different values for the flow field: * "client" indicates that this vantage point follows client data flow semantics (a "packet sent" event goes in the direction of the server). * "server" indicates that this vantage point follow server data flow semantics (a "packet sent" event goes in the direction of the client). * "unknown" indicates that the flow's direction is unknown. Depending on the context, tools confronted with "unknown" values in the vantage_point can either try to heuristically infer the semantics from protocol-level domain knowledge (e.g., in QUIC, the client always sends the first packet) or give the user the option to switch between client and server perspectives manually.

Field name semantics Inside of the "events" field of a qlog trace is a list of events logged by the endpoint. Each event is specified as a generic object with a number of member fields and their associated data. Depending on the protocol and use case, the exact member field names and their formats can differ across implementations. This section lists the main, pre-defined and reserved field names with specific semantics and expected corresponding value formats. Each qlog event at minimum requires the "time" (), "name" () and "data" () fields. Other typical fields are "time_format" (), "protocol_type" (), "trigger" (), and "group_id" . As especially these later fields typically have identical values across individual event instances, they are normally logged separately in the "common_fields" (). The specific values for each of these fields and their semantics are defined in separate documents, specific per protocol or use case. For example: event definitions for QUIC, HTTP/3 and QPACK can be found in and . Other fields are explicitly allowed by the qlog approach, and tools SHOULD allow for the presence of unknown event fields, but their semantics depend on the context of the log usage (e.g., for QUIC, the ODCID field is used), see . An example of a qlog event with its component fields is shown in . Definition:

Event definition any } ]]>

JSON serialization:

Event example

Timestamps The "time" field indicates the timestamp at which the event occurred. Its value is typically the Unix timestamp since the 1970 epoch (number of milliseconds since midnight UTC, January 1, 1970, ignoring leap seconds). However, qlog supports two more succinct timestamps formats to allow reducing file size. The employed format is indicated in the "time_format" field, which allows one of three values: "absolute", "delta" or "relative". Definition:

TimeFormat definition

• Absolute: Include the full absolute timestamp with each event. This approach uses the largest amount of characters. This is also the default value of the "time_format" field.
• Delta: Delta-encode each time value on the previously logged value. The first event in a trace typically logs the full absolute timestamp. This approach uses the least amount of characters.
• Relative: Specify a full "reference_time" timestamp (typically this is done up-front in "common_fields", see ) and include only relatively-encoded values based on this reference_time with each event. The "reference_time" value is typically the first absolute timestamp. This approach uses a medium amount of characters.

The first option is good for stateless loggers, the second and third for stateful loggers. The third option is generally preferred, since it produces smaller files while being easier to reason about. An example for each option can be seen in .

Three different approaches for logging timestamps

One of these options is typically chosen for the entire trace (put differently: each event has the same value for the "time_format" field). Each event MUST include a timestamp in the "time" field. Events in each individual trace SHOULD be logged in strictly ascending timestamp order (though not necessarily absolute value, for the "delta" format). Tools CAN sort all events on the timestamp before processing them, though are not required to (as this could impose a significant processing overhead). This can be a problem especially for multi-threaded and/or streaming loggers, who could consider using a separate post-processor to order qlog events in time if a tool do not provide this feature. Timestamps do not have to use the UNIX epoch timestamp as their reference. For example for privacy considerations, any initial reference timestamps (for example "endpoint uptime in ms" or "time since connection start in ms") can be chosen. Tools SHOULD NOT assume the ability to derive the absolute Unix timestamp from qlog traces, nor allow on them to relatively order events across two or more separate traces (in this case, clock drift should also be taken into account).

Category and Event Type Events differ mainly in the type of metadata associated with them. To help identify a given event and how to interpret its metadata in the "data" field (see ), each event has an associated "name" field. This can be considered as a concatenation of two other fields, namely event "category" and event "type". Category allows a higher-level grouping of events per specific event type. For example for QUIC and HTTP/3, the different categories could be "transport", "http", "qpack", and "recovery". Within these categories, the event Type provides additional granularity. For example for QUIC and HTTP/3, within the "transport" Category, there would be "packet_sent" and "packet_received" events. Logging category and type separately conceptually allows for fast and high-level filtering based on category and the re-use of event types across categories. However, it also considerably inflates the log size and this flexibility is not used extensively in practice at the time of writing. As such, the default approach in qlog is to concatenate both field values using the ":" character in the "name" field, as can be seen in . As such, qlog category and type names MUST NOT include this character.

Ways of logging category, type and name of an event.

Certain serializations CAN emit category and type as separate fields, and qlog tools SHOULD be able to deal with both the concatenated "name" field, and the separate "category" and "type" fields. Text-based serializations however are encouraged to employ the concatenated "name" field for efficiency.

Data The data field is a generic object. It contains the per-event metadata and its form and semantics are defined per specific sort of event. For example, data field value definitions for QUIC and HTTP/3 can be found in and . This field is defined here as a CDDL extension point (a "socket" or "plug") named $ProtocolEventBody. Other documents MUST properly extend this extension point when defining new data field content options to enable automated validation of aggregated qlog schemas. The only common field defined for the data field is the trigger field, which is discussed in . Definition:

ProtocolEventBody definition any } ; event documents are intended to extend this socket by using: ; NewProtocolEvents = EventType1 / EventType2 / ... / EventTypeN ; $ProtocolEventBody /= NewProtocolEvents ]]>

One purely illustrative example for a QUIC "packet_sent" event is shown in :

Example of the 'data' field for a QUIC packet_sent event

protocol_type The "protocol_type" array field indicates to which protocols (or protocol "stacks") this event belongs. This allows a single qlog file to aggregate traces of different protocols (e.g., a web server offering both TCP+HTTP/2 and QUIC+HTTP/3 connections). Definition:

ProtocolType definition

For example, QUIC and HTTP/3 events have the "QUIC" and "HTTP3" protocol_type entry values, see and . Typically however, all events in a single trace are of the same few protocols, and this array field is logged once in "common_fields", see .

Triggers Sometimes, additional information is needed in the case where a single event can be caused by a variety of other events. In the normal case, the context of the surrounding log messages gives a hint as to which of these other events was the cause. However, in highly-parallel and optimized implementations, corresponding log messages might separated in time. Another option is to explicitly indicate these "triggers" in a high-level way per-event to get more fine-grained information without much additional overhead. In qlog, the optional "trigger" field contains a string value describing the reason (if any) for this event instance occurring, see . While this "trigger" field could be a property of the qlog Event itself, it is instead a property of the "data" field instead. This choice was made because many event types do not include a trigger value, and having the field at the Event-level would cause overhead in some serializations. Additional information on the trigger can be added in the form of additional member fields of the "data" field value, yet this is highly implementation-specific, as are the trigger field's string values. One purely illustrative example of some potential triggers for QUIC's "packet_dropped" event is shown in :

Trigger example

group_id As discussed in , a single qlog file can contain several traces taken from different vantage points. However, a single trace from one endpoint can also contain events from a variety of sources. For example, a server implementation might choose to log events for all incoming connections in a single large (streamed) qlog file. As such, we need a method for splitting up events belonging to separate logical entities. The simplest way to perform this splitting is by associating a "group identifier" to each event that indicates to which conceptual "group" each event belongs. A post-processing step can then extract events per group. However, this group identifier can be highly protocol and context-specific. In the example above, we might use QUIC's "Original Destination Connection ID" to uniquely identify a connection. As such, they might add a "ODCID" field to each event. However, a middlebox logging IP or TCP traffic might rather use four-tuples to identify connections, and add a "four_tuple" field. As such, to provide consistency and ease of tooling in cross-protocol and cross-context setups, qlog instead defines the common "group_id" field, which contains a string value. Implementations are free to use their preferred string serialization for this field, so long as it contains a unique value per logical group. Some examples can be seen in . Definition:

GroupID definition

JSON serialization example for events grouped by four tuples and QUIC connection IDs:

GroupID example

Note that in some contexts (for example a Multipath transport protocol) it might make sense to add additional contextual per-event fields (for example "path_id"), rather than use the group_id field for that purpose. Note also that, typically, a single trace only contains events belonging to a single logical group (for example, an individual QUIC connection). As such, instead of logging the "group_id" field with an identical value for each event instance, this field is typically logged once in "common_fields", see .

common_fields As discussed in the previous sections, information for a typical qlog event varies in three main fields: "time", "name" and associated data. Additionally, there are also several more advanced fields that allow mixing events from different protocols and contexts inside of the same trace (for example "protocol_type" and "group_id"). In most "normal" use cases however, the values of these advanced fields are consistent for each event instance (for example, a single trace contains events for a single QUIC connection). To reduce file size and making logging easier, qlog uses the "common_fields" list to indicate those fields and their values that are shared by all events in this component trace. This prevents these fields from being logged for each individual event. An example of this is shown in .

CommonFields example

The "common_fields" field is a generic dictionary of key-value pairs, where the key is always a string and the value can be of any type, but is typically also a string or number. As such, unknown entries in this dictionary MUST be disregarded by the user and tools (i.e., the presence of an unknown field is explicitly NOT an error). The list of default qlog fields that are typically logged in common_fields (as opposed to as individual fields per event instance) are shown in the listing below: Definition:

CommonFields definition any } ]]>

Tools MUST be able to deal with these fields being defined either on each event individually or combined in common_fields. Note that if at least one event in a trace has a different value for a given field, this field MUST NOT be added to common_fields but instead defined on each event individually. Good example of such fields are "time" and "data", who are divergent by nature.

Guidelines for event definition documents This document only defines the main schema for the qlog format. This is intended to be used together with specific, per-protocol event definitions that specify the name (category + type) and data needed for each individual event. This is with the intent to allow the qlog main schema to be easily re-used for several protocols. Examples include the QUIC event definitions and HTTP/3 and QPACK event definitions . This section defines some basic annotations and concepts the creators of event definition documents SHOULD follow to ensure a measure of consistency, making it easier for qlog implementers to extrapolate from one protocol to another.

Event design guidelines TODO: pending QUIC working group discussion. This text reflects the initial (qlog draft 01 and 02) setup. There are several ways of defining qlog events. In practice, we have seen two main types used so far: a) those that map directly to concepts seen in the protocols (e.g., packet_sent) and b) those that act as aggregating events that combine data from several possible protocol behaviors or code paths into one (e.g., parameters_set). The latter are typically used as a means to reduce the amount of unique event definitions, as reflecting each possible protocol event as a separate qlog entity would cause an explosion of event types. Additionally, logging duplicate data is typically prevented as much as possible. For example, packet header values that remain consistent across many packets are split into separate events (for example spin_bit_updated or connection_id_updated for QUIC). Finally, we have typically refrained from adding additional state change events if those state changes can be directly inferred from data on the wire (for example flow control limit changes) if the implementation is bug-free and spec-compliant. Exceptions have been made for common events that benefit from being easily identifiable or individually logged (for example packets_acked).

Event importance indicators Depending on how events are designed, it may be that several events allow the logging of similar or overlapping data. For example the separate QUIC connection_started event overlaps with the more generic connection_state_updated. In these cases, it is not always clear which event should be logged or used, and which event should take precedence if e.g., both are present and provide conflicting information. To aid in this decision making, we recommend that each event SHOULD have an "importance indicator" with one of three values, in decreasing order of importance and expected usage:

• Core
• Base
• Extra

The "Core" events are the events that SHOULD be present in all qlog files for a given protocol. These are typically tied to basic packet and frame parsing and creation, as well as listing basic internal metrics. Tool implementers SHOULD expect and add support for these events, though SHOULD NOT expect all Core events to be present in each qlog trace. The "Base" events add additional debugging options and CAN be present in qlog files. Most of these can be implicitly inferred from data in Core events (if those contain all their properties), but for many it is better to log the events explicitly as well, making it clearer how the implementation behaves. These events are for example tied to passing data around in buffers, to how internal state machines change and help show when decisions are actually made based on received data. Tool implementers SHOULD at least add support for showing the contents of these events, if they do not handle them explicitly. The "Extra" events are considered mostly useful for low-level debugging of the implementation, rather than the protocol. They allow more fine-grained tracking of internal behavior. As such, they CAN be present in qlog files and tool implementers CAN add support for these, but they are not required to. Note that in some cases, implementers might not want to log for example data content details in the "Core" events due to performance or privacy considerations. In this case, they SHOULD use (a subset of) relevant "Base" events instead to ensure usability of the qlog output. As an example, implementations that do not log QUIC packet_received events and thus also not which (if any) ACK frames the packet contains, SHOULD log packets_acked events instead. Finally, for event types whose data (partially) overlap with other event types' definitions, where necessary the event definition document should include explicit guidance on which to use in specific situations.

Custom fields Event definition documents are free to define new category and event types, top-level fields (e.g., a per-event field indicating its privacy properties or path_id in multipath protocols), as well as values for the "trigger" property within the "data" field, or other member fields of the "data" field, as they see fit. They however SHOULD NOT expect non-specialized tools to recognize or visualize this custom data. However, tools SHOULD make an effort to visualize even unknown data if possible in the specific tool's context. If they do not, they MUST ignore these unknown fields.

Generic events and data classes There are some event types and data classes that are common across protocols, applications and use cases that benefit from being defined in a single location. This section specifies such common definitions.

Raw packet and frame information While qlog is a more high-level logging format, it also allows the inclusion of most raw wire image information, such as byte lengths and even raw byte values. This can be useful when for example investigating or tuning packetization behavior or determining encoding/framing overheads. However, these fields are not always necessary and can take up considerable space if logged for each packet or frame. They can also have a considerable privacy and security impact. As such, they are grouped in a separate optional field called "raw" of type RawInfo (where applicable). Definition:

RawInfo definition

Note:

The RawInfo:data field can be truncated for privacy or security purposes (for example excluding payload data), see . In this case, the length properties should still indicate the non-truncated lengths.

Note:

We do not specify explicit header_length or trailer_length fields. In most protocols, header_length can be calculated by subtracting the payload_length from the length (e.g., if trailer_length is always 0). In protocols with trailers (e.g., QUIC's AEAD tag), event definitions documents SHOULD define other ways of logging the trailer_length to make the header_length calculation possible.

The exact definitions entities, headers, trailers and payloads depend on the protocol used. If this is non-trivial, event definitions documents SHOULD include a clear explanation of how entities are mapped into the RawInfo structure.

Note:

Relatedly, many modern protocols use Variable-Length Integer Encoded (VLIE) values in their headers, which are of a dynamic length. Because of this, we cannot deterministically reconstruct the header encoding/length from non-RawInfo qlog data, as implementations might not necessarily employ the most efficient VLIE scheme for all values. As such, to make exact size-analysis possible, implementers should use explicit lengths in RawInfo rather than reconstructing them from other qlog data. Similarly, tool developers should only utilize RawInfo (and related information) in such tools to prevent errors.

Generic events In typical logging setups, users utilize a discrete number of well-defined logging categories, levels or severities to log freeform (string) data. This generic events category replicates this approach to allow implementations to fully replace their existing text-based logging by qlog. This is done by providing events to log generic strings for the typical well-known logging levels (error, warning, info, debug, verbose). For the events defined below, the "category" is "generic" and their "type" is the name of the heading in lowercase (e.g., the "name" of the error event is "generic:error").

error Importance: Core Used to log details of an internal error that might not get reflected on the wire. Definition:

GenericError definition

warning Importance: Base Used to log details of an internal warning that might not get reflected on the wire. Definition:

GenericWarning definition

info Importance: Extra Used mainly for implementations that want to use qlog as their one and only logging format but still want to support unstructured string messages. Definition:

GenericInfo definition

debug Importance: Extra Used mainly for implementations that want to use qlog as their one and only logging format but still want to support unstructured string messages. Definition:

GenericDebug definition

verbose Importance: Extra Used mainly for implementations that want to use qlog as their one and only logging format but still want to support unstructured string messages. Definition:

GenericVerbose definition

Simulation events When evaluating a protocol implementation, one typically sets up a series of interoperability or benchmarking tests, in which the test situations can change over time. For example, the network bandwidth or latency can vary during the test, or the network can be fully disable for a short time. In these setups, it is useful to know when exactly these conditions are triggered, to allow for proper correlation with other events. For the events defined below, the "category" is "simulation" and their "type" is the name of the heading in lowercase (e.g., the "name" of the scenario event is "simulation:scenario").

scenario Importance: Extra Used to specify which specific scenario is being tested at this particular instance. This could also be reflected in the top-level qlog's summary or configuration fields, but having a separate event allows easier aggregation of several simulations into one trace (e.g., split by group_id). Definition:

SimulationScenario definition any } } ]]>

marker Importance: Extra Used to indicate when specific emulation conditions are triggered at set times (e.g., at 3 seconds in 2% packet loss is introduced, at 10s a NAT rebind is triggered). Definition:

SimulationMarker definition

Serializing qlog This document and other related qlog schema definitions are intentionally serialization-format agnostic. This means that implementers themselves can choose how to represent and serialize qlog data practically on disk or on the wire. Some examples of possible formats are JSON, CBOR, CSV, protocol buffers, flatbuffers, etc. All these formats make certain tradeoffs between flexibility and efficiency, with textual formats like JSON typically being more flexible but also less efficient than binary formats like protocol buffers. The format choice will depend on the practical use case of the qlog user. For example, for use in day to day debugging, a plaintext readable (yet relatively large) format like JSON is probably preferred. However, for use in production, a more optimized yet restricted format can be better. In this latter case, it will be more difficult to achieve interoperability between qlog implementations of various protocol stacks, as some custom or tweaked events from one might not be compatible with the format of the other. This will also reflect in tooling: not all tools will support all formats. This being said, the authors prefer JSON as the basis for storing qlog, as it retains full flexibility and maximum interoperability. Storage overhead can be managed well in practice by employing compression. For this reason, this document details how to practically transform qlog schema definitions to , its subset , and its streamable derivative s. We discuss concrete options to bring down JSON size and processing overheads in . As depending on the employed format different deserializers/parsers should be used, the "qlog_format" field is used to indicate the chosen serialization approach. This field is always a string, but can be made hierarchical by the use of the "." separator between entries. For example, a value of "JSON.optimizationA" can indicate that a default JSON format is being used, but that a certain optimization of type A was applied to the file as well (see also ).

qlog to JSON mapping When mapping qlog to normal JSON, the "qlog_format" field MUST have the value "JSON". This is also the default qlog serialization and default value of this field. When using normal JSON serialization, the file extension/suffix SHOULD be ".qlog" and the Media Type (if any) SHOULD be "application/qlog+json" per . JSON files by definition () MUST utilize the UTF-8 encoding, both for the file itself and the string values. While not specifically required by the JSON specification, all qlog field names in a JSON serialization MUST be lowercase. In order to serialize CDDL-based qlog event and data structure definitions to JSON, the official CDDL-to-JSON mapping defined in Appendix E of SHOULD be employed.

I-JSON For some use cases, it should be taken into account that not all popular JSON parsers support the full JSON format. Especially for parsers integrated with the JavaScript programming language (e.g., Web browsers, NodeJS), users are recommended to stick to a JSON subset dubbed (or Internet-JSON). One of the key limitations of JavaScript and thus I-JSON is that it cannot represent full 64-bit integers in standard operating mode (i.e., without using BigInt extensions), instead being limited to the range of [-(2**53)+1, (2**53)-1]. In these circumstances, Appendix E of recommends defining new CDDL types for int64 and uint64 that limit their values to this range. While this can be sensible and workable for most use cases, some protocols targeting qlog serialization (e.g., QUIC, HTTP/3), might require full uint64 variables in some (rare) circumstances. In these situations, it should be allowed to also use the string-based representation of uint64 values alongside the numerical representation. Concretely, the following definition of uint64 should override the original and (web-based) tools should take into account that a uint64 field can be either a number or string.

Custom uint64 definition for I-JSON

Truncated values For some use cases (e.g., limiting file size, privacy), it can be necessary not to log a full raw blob (using the hexstring type) but instead a truncated value (for example, only the first 100 bytes of an HTTP response body to be able to discern which file it actually contained). In these cases, the original byte-size length cannot be obtained from the serialized value directly. As such, all qlog schema definitions SHOULD include a separate, length-indicating field for all fields of type hexstring they specify, see for example . This not only ensures the original length can always be retrieved, but also allows the omission of any raw value bytes of the field completely (e.g., out of privacy or security considerations). To reduce overhead however and in the case the full raw value is logged, the extra length-indicating field can be left out. As such, tools MUST be able to deal with this situation and derive the length of the field from the raw value if no separate length-indicating field is present. The main possible permutations are shown by example in .

Example for serializing truncated hexstrings

qlog to JSON Text Sequences mapping One of the downsides of using pure JSON is that it is inherently a non-streamable format. Put differently, it is not possible to simply append new qlog events to a log file without "closing" this file at the end by appending "]}]}". Without these closing tags, most JSON parsers will be unable to parse the file entirely. As most platforms do not provide a standard streaming JSON parser (which would be able to deal with this problem), this document also provides a qlog mapping to a streamable JSON format called JSON Text Sequences (JSON-SEQ) (). When mapping qlog to JSON-SEQ, the "qlog_format" field MUST have the value "JSON-SEQ". When using JSON-SEQ serialization, the file extension/suffix SHOULD be ".sqlog" (for "streaming" qlog) and the Media Type (if any) SHOULD be "application/qlog+json-seq" per . JSON Text Sequences are very similar to JSON, except that JSON objects are serialized as individual records, each prefixed by an ASCII Record Separator (<RS>, 0x1E), and each ending with an ASCII Line Feed character (\n, 0x0A). Note that each record can also contain any amount of newlines in its body, as long as it ends with a newline character before the next <RS> character. Each qlog event is serialized and interpreted as an individual JSON Text Sequence record, and can simply be appended as a new object at the back of an event stream or log file. Put differently, unlike default JSON, it does not require a file to be wrapped as a full object with "{ ... }" or "[... ]". For this to work, some qlog definitions have to be adjusted however. Mainly, events are no longer part of the "events" array in the Trace object, but are instead logged separately from the qlog "header", as indicated by the TraceSeq object in . Additionally, qlog's JSON-SEQ mapping does not allow logging multiple individual traces in a single qlog file. As such, the QlogFile:traces field is replaced by the singular QlogFileSeq:trace field, see . An example can be seen in . Note that the "group_id" field can still be used on a per-event basis to include events from conceptually different sources in a single JSON-SEQ qlog file. Definition:

TraceSeq definition

Definition:

QlogFileSeq definition

JSON-SEQ serialization examples:

Top-level element { "qlog_version": "0.3", "qlog_format": "JSON-SEQ", "title": "Name of JSON Text Sequence qlog file (short)", "description": "Description for this trace file (long)", "summary": { ... }, "trace": { "common_fields": { "protocol_type": ["QUIC","HTTP3"], "group_id":"127ecc830d98f9d54a42c4f0842aa87e181a", "time_format":"relative", "reference_time": 1553986553572 }, "vantage_point": { "name":"backend-67", "type":"server" } } } {"time": 2, "name": "transport:parameters_set", "data": { ... } } {"time": 7, "name": "transport:packet_sent", "data": { ... } } ... ]]>

Note: while not specifically required by the JSON-SEQ specification, all qlog field names in a JSON-SEQ serialization MUST be lowercase. In order to serialize all other CDDL-based qlog event and data structure definitions to JSON-SEQ, the official CDDL-to-JSON mapping defined in Appendix E of SHOULD still be employed.

Supporting JSON Text Sequences in tooling Note that JSON Text Sequences are not supported in most default programming environments (unlike normal JSON). However, several custom JSON-SEQ parsing libraries exist in most programming languages that can be used and the format is easy enough to parse with existing implementations (i.e., by splitting the file into its component records and feeding them to a normal JSON parser individually, as each record by itself is a valid JSON object).

Other optimized formatting options Both the JSON and JSON-SEQ formatting options described above are serviceable in general small to medium scale (debugging) setups. However, these approaches tend to be relatively verbose, leading to larger file sizes. Additionally, generalized JSON(-SEQ) (de)serialization performance is typically (slightly) lower than that of more optimized and predictable formats. Both aspects make these formats more challenging (though still practical) to use in large scale setups. During the development of qlog, we compared a multitude of alternative formatting and optimization options. The results of this study are summarized on the qlog github repository. The rest of this section discusses some of these approaches implementations could choose and the expected gains and tradeoffs inherent therein. Tools SHOULD support mainly the compression options listed in , as they provide the largest wins for the least cost overall. Over time, specific qlog formats and encodings can be created that more formally define and combine some of the discussed optimizations or add new ones. We choose to define these schemes in separate documents to keep the main qlog definition clean and generalizable, as not all contexts require the same performance or flexibility as others and qlog is intended to be a broadly usable and extensible format (for example more flexibility is needed in earlier stages of protocol development, while more performance is typically needed in later stages). This is also the main reason why the general qlog format is the less optimized JSON instead of a more performant option. To be able to easily distinguish between these options in qlog compatible tooling (without the need to have the user provide out-of-band information or to (heuristically) parse and process files in a multitude of ways, see also ), we recommend using explicit file extensions to indicate specific formats. As there are no standards in place for this type of extension to format mapping, we employ a commonly used scheme here. Our approach is to list the applied optimizations in the extension in ascending order of application (e.g., if a qlog file is first optimized with technique A and then compressed with technique B, the resulting file would have the extension ".(s)qlog.A.B"). This allows tooling to start at the back of the extension to "undo" applied optimizations to finally arrive at the expected qlog representation.

Data structure optimizations The first general category of optimizations is to alter the representation of data within an JSON(-SEQ) qlog file to reduce file size. The first option is to employ a scheme similar to the CSV (comma separated value ) format, which utilizes the concept of column "headers" to prevent repeating field names for each datapoint instance. Concretely for JSON qlog, several field names are repeated with each event (i.e., time, name, data). These names could be extracted into a separate list, after which qlog events could be serialized as an array of values, as opposed to a full object. This approach was a key part of the original qlog format (prior to draft-02) using the "event_fields" field. However, tests showed that this optimization only provided a mean file size reduction of 5% (100MB to 95MB) while significantly increasing the implementation complexity, and this approach was abandoned in favor of the default JSON setup. Implementations using this format should not employ a separate file extension (as it still uses JSON), but rather employ a new value of "JSON.namedheaders" (or "JSON-SEQ.namedheaders") for the "qlog_format" field (see ). The second option is to replace field values and/or names with indices into a (dynamic) lookup table. This is a common compression technique and can provide significant file size reductions (up to 50% in our tests, 100MB to 50MB). However, this approach is even more difficult to implement efficiently and requires either including the (dynamic) table in the resulting file (an approach taken by for example Chromium's NetLog format) or defining a (static) table up-front and sharing this between implementations. Implementations using this approach should not employ a separate file extension (as it still uses JSON), but rather employ a new value of "JSON.dictionary" (or "JSON-SEQ.dictionary") for the "qlog_format" field (see ). As both options either proved difficult to implement, reduced qlog file readability, and provided too little improvement compared to other more straightforward options (for example ), these schemes are not inherently part of qlog.

Compression The second general category of optimizations is to utilize a (generic) compression scheme for textual data. As qlog in the JSON(-SEQ) format typically contains a large amount of repetition, off-the-shelf (text) compression techniques typically succeed very well in bringing down file sizes (regularly with up to two orders of magnitude in our tests, even for "fast" compression levels). As such, utilizing compression is recommended before attempting other optimization options, even though this might (somewhat) increase processing costs due to the additional compression step. The first option is to use GZIP compression (). This generic compression scheme provides multiple compression levels (providing a trade-off between compression speed and size reduction). Utilized at level 6 (a medium setting thought to be applicable for streaming compression of a qlog stream in commodity devices), gzip compresses qlog JSON files to 7% of their initial size on average (100MB to 7MB). For this option, the file extension .(s)qlog.gz SHOULD BE used. The "qlog_format" field should still reflect the original JSON formatting of the qlog data (e.g., "JSON" or "JSON-SEQ"). The second option is to use Brotli compression (). While similar to gzip, this more recent compression scheme provides a better efficiency. It also allows multiple compression levels. Utilized at level 4 (a medium setting thought to be applicable for streaming compression of a qlog stream in commodity devices), brotli compresses qlog JSON files to 7% of their initial size on average (100MB to 7MB). For this option, the file extension .(s)qlog.br SHOULD BE used. The "qlog_format" field should still reflect the original JSON formatting of the qlog data (e.g., "JSON" or "JSON-SEQ"). Other compression algorithms of course exist (for example xz, zstd, and lz4). We mainly recommend gzip and brotli because of their tweakable behaviour and wide support in web-based environments, which we envision as the main tooling ecosystem (see also ).

Binary formats The third general category of optimizations is to use a more optimized (often binary) format instead of the textual JSON format. This approach inherently produces smaller files and often has better (de)serialization performance. However, the resultant files are no longer human readable and some formats require hard tradeoffs between flexibility for performance. The first option is to use the CBOR (Concise Binary Object Representation ) format. For our purposes, CBOR can be viewed as a straightforward binary variant of JSON. As such, existing JSON qlog files can be trivially converted to and from CBOR (though slightly more work is needed for JSON-SEQ qlogs to convert them to CBOR-SEQ, see ). While CBOR thus does retain the full qlog flexibility, it only provides a 25% file size reduction (100MB to 75MB) compared to textual JSON(-SEQ). As CBOR support in programming environments is not as widespread as that of textual JSON and the format lacks human readability, CBOR was not chosen as the default qlog format. For this option, the file extension .(s)qlog.cbor SHOULD BE used. The "qlog_format" field should still reflect the original JSON formatting of the qlog data (e.g., "JSON" or "JSON-SEQ"). The media type should indicate both whether JSON or JSON Text Sequences are used, as well as whether CBOR or CBOR Sequences are used (see the table below). A second option is to use a more specialized binary format, such as Protocol Buffers (protobuf). This format is battle-tested, has support for optional fields and has libraries in most programming languages. Still, it is significantly less flexible than textual JSON or CBOR, as it relies on a separate, pre-defined schema (a .proto file). As such, it it not possible to (easily) log new event types in protobuf files without adjusting this schema as well, which has its own practical challenges. As qlog is intended to be a flexible, general purpose format, this type of format was not chosen as its basic serialization. The lower flexibility does lead to significantly reduced file sizes. Our straightforward mapping of the qlog main schema and QUIC/HTTP3 event types to protobuf created qlog files 24% as large as the raw JSON equivalents (100MB to 24MB). For this option, the file extension .(s)qlog.protobuf SHOULD BE used. The "qlog_format" field should reflect the different internal format, for example: "qlog_format": "protobuf". Note that binary formats can (and should) also be used in conjunction with compression (see ). For example, CBOR compresses well (to about 6% of the original textual JSON size (100MB to 6MB) for both gzip and brotli) and so does protobuf (5% (gzip) to 3% (brotli)). However, these gains are similar to the ones achieved by simply compression the textual JSON equivalents directly (7%, see ). As such, since compression is still needed to achieve optimal file size reductions event with binary formats, we feel the more flexible compressed textual JSON options are a better default for the qlog format in general.

Overview and summary In summary, textual JSON was chosen as the main qlog format due to its high flexibility and because its inefficiencies can be largely solved by the utilization of compression techniques (which are needed to achieve optimal results with other formats as well). Still, qlog implementers are free to define other qlog formats depending on their needs and context of use. These formats should be described in their own documents, the discussion in this document mainly acting as inspiration and high-level guidance. Implementers are encouraged to add concrete qlog formats and definitions to the designated public repository. The following table provides an overview of all the discussed qlog formatting options with examples:

format	qlog_format	extension	media type
JSON	JSON	.qlog	application/qlog+json
JSON Text Sequences	JSON-SEQ	.sqlog	application/qlog+json-seq
named headers	JSON(-SEQ).namedheaders	.(s)qlog	application/qlog+json(-seq)
dictionary	JSON(-SEQ).dictionary	.(s)qlog	application/qlog+json(-seq)
CBOR	JSON(-SEQ)	.(s)qlog.cbor	application/qlog+json(-seq)+cbor(-seq)
protobuf	protobuf	.qlog.protobuf	NOT SPECIFIED BY IANA
gzip	no change	.gz suffix	application/gzip
brotli	no change	.br suffix	NOT SPECIFIED BY IANA

Conversion between formats As discussed in the previous sections, a qlog file can be serialized in a multitude of formats, each of which can conceivably be transformed into or from one another without loss of information. For example, a number of JSON-SEQ streamed qlogs could be combined into a JSON formatted qlog for later processing. Similarly, a captured binary qlog could be transformed to JSON for easier interpretation and sharing. Secondly, we can also consider other structured logging approaches that contain similar (though typically not identical) data to qlog, like raw packet capture files (for example .pcap files from tcpdump) or endpoint-specific logging formats (for example the NetLog format in Google Chrome). These are sometimes the only options, if an implementation cannot or will not support direct qlog output for any reason, but does provide other internal or external (e.g., SSLKEYLOGFILE export to allow decryption of packet captures) logging options For this second category, a (partial) transformation from/to qlog can also be defined. As such, when defining a new qlog serialization format or wanting to utilize qlog-compatible tools with existing codebases lacking qlog support, it is recommended to define and provide a concrete mapping from one format to default JSON-serialized qlog. Several of such mappings exist. Firstly, [pcap2qlog]((https://github.com/quiclog/pcap2qlog) transforms QUIC and HTTP/3 packet capture files to qlog. Secondly, netlog2qlog converts chromium's internal dictionary-encoded JSON format to qlog. Finally, quictrace2qlog converts the older quictrace format to JSON qlog. Tools can then easily integrate with these converters (either by incorporating them directly or for example using them as a (web-based) API) so users can provide different file types with ease. For example, the qvis toolsuite supports a multitude of formats and qlog serializations.

Methods of access and generation Different implementations will have different ways of generating and storing qlogs. However, there is still value in defining a few default ways in which to steer this generation and access of the results.

Set file output destination via an environment variable To provide users control over where and how qlog files are created, we define two environment variables. The first, QLOGFILE, indicates a full path to where an individual qlog file should be stored. This path MUST include the full file extension. The second, QLOGDIR, sets a general directory path in which qlog files should be placed. This path MUST include the directory separator character at the end. In general, QLOGDIR should be preferred over QLOGFILE if an endpoint is prone to generate multiple qlog files. This can for example be the case for a QUIC server implementation that logs each QUIC connection in a separate qlog file. An alternative that uses QLOGFILE would be a QUIC server that logs all connections in a single file and uses the "group_id" field () to allow post-hoc separation of events. Implementations SHOULD provide support for QLOGDIR and MAY provide support for QLOGFILE. When using QLOGDIR, it is up to the implementation to choose an appropriate naming scheme for the qlog files themselves. The chosen scheme will typically depend on the context or protocols used. For example, for QUIC, it is recommended to use the Original Destination Connection ID (ODCID), followed by the vantage point type of the logging endpoint. Examples of all options for QUIC are shown in .

Environment variable examples for a QUIC implementation

Access logs via a well-known endpoint After generation, qlog implementers MAY make available generated logs and traces on an endpoint (typically the server) via the following .well-known URI:

• .well-known/qlog/IDENTIFIER.extension

The IDENTIFIER variable depends on the context and the protocol. For example for QUIC, the lowercase Original Destination Connection ID (ODCID) is recommended, as it can uniquely identify a connection. Additionally, the extension depends on the chosen format (see ). For example, for a QUIC connection with ODCID "abcde", the endpoint for fetching its default JSON-formatted .qlog file would be:

• .well-known/qlog/abcde.qlog

Implementers SHOULD allow users to fetch logs for a given connection on a 2nd, separate connection. This helps prevent pollution of the logs by fetching them over the same connection that one wishes to observe through the log. Ideally, for the QUIC use case, the logs should also be approachable via an HTTP/2 or HTTP/1.1 endpoint (i.e., on TCP port 443), to for example aid debugging in the case where QUIC/UDP is blocked on the network. qlog implementers SHOULD NOT enable this .well-known endpoint in typical production settings to prevent (malicious) users from downloading logs from other connections. Implementers are advised to disable this endpoint by default and require specific actions from the end users to enable it (and potentially qlog itself). Implementers MUST also take into account the general privacy and security guidelines discussed in before exposing qlogs to outside actors.

Tooling requirements Tools ingestion qlog MUST indicate which qlog version(s), qlog format(s), compression methods and potentially other input file formats (for example .pcap) they support. Tools SHOULD at least support .qlog files in the default JSON format (). Additionally, they SHOULD indicate exactly which values for and properties of the name (category and type) and data fields they look for to execute their logic. Tools SHOULD perform a (high-level) check if an input qlog file adheres to the expected qlog schema. If a tool determines a qlog file does not contain enough supported information to correctly execute the tool's logic, it SHOULD generate a clear error message to this effect. Tools MUST NOT produce breaking errors for any field names and/or values in the qlog format that they do not recognize. Tools SHOULD indicate even unknown event occurrences within their context (e.g., marking unknown events on a timeline for manual interpretation by the user). Tool authors should be aware that, depending on the logging implementation, some events will not always be present in all traces. For example, using a circular logging buffer of a fixed size, it could be that the earliest events (e.g., connection setup events) are later overwritten by "newer" events. Alternatively, some events can be intentionally omitted out of privacy or file size considerations. Tool authors are encouraged to make their tools robust enough to still provide adequate output for incomplete logs.

Security and privacy considerations Protocols such as TLS and QUIC provide varying degrees of secure protection for the wire image . There is inevitably tension between security and observability, when logging can reveal aspects of the wire image, that would ordinarily be protected. This tension equally applies to any privacy considerations that build on security properties, especially if data can be correlated across data sources. qlog operators and implementers should be mindful of the security and privacy risks inherent in handling qlog data. This includes but is not limited to logging, storing, or using the data. Data might be considered as non-sensitive, potentially-sensitive, or sensitive; applying the considerations in this section may produce different risks depending on the nature of the data itself, or its handling. However, in many cases the largest risk factors arise from data that can be considered as potenially-sensitive or sensitive. The following is a non-exhaustive list of such fields and types of data that can be carried in qlog data:

• IP addresses and transport protocol port numbers, which can be used to uniquely identify individual connections, endpoints, and potentially users.
• Session, Connection, or User identifiers which can be used to correlate nominally separate contexts. For example, QUIC Connection IDs can be used to identify and track users across geographical networks ).
• Stored State which can be used to correlate individual connections or sessions over time. Examples include QUIC address validation and retry tokens, TLS session tickets, and HTTP cookies.
• Decryption keys, passwords, and tokens which can be used with other data sources (e.g., captures of encrypted packets) to correlate qlog data to a specific connection or user or leak additional information. Examples include TLS decryption keys and HTTP-level API access or authorization tokens.
• Data that can be used to correlate qlogs to other data sources (e.g., captures of encrypted packets). Examples include high-resolution event timestamps or inter-event timings, event counts, packet and frame sizes.
• Full or partial encrypted raw packet and frame payloads, which can be used with other data sources (e.g., captures of encrypted packets) to correlate qlog data to a specific connection or session.
• Full or partial plaintext raw packet and frame payloads (e.g., HTTP Field values, HTTP response data, TLS SNI field values), which can contain directly sensitive information.

The simplest and most extreme form of protection against abuse of this information is the complete deletion of a given field, which is equivalent to not logging the field(s) in question. While deletion completely protects the data in the deleted fields from the risk of compromise, it also reduces the utility of the dataset as a whole. As such, a balance should be found between logging these fields and the potential risks inherent in their (involuntary) disclosure. This balance depends on the use case at hand (e.g., research datasets might have different requirements to live operational troubleshooting). Capturing the minimal amount of data required for a specific purpose can help to minimize the risks associated with data usage. qlog implementations that provide fine-grained control over the inclusion of data fields, ideally on a per-use-case or per-connection basis, improve the ability to minimize data. Any data that is determined to be necessary for a use case at hand could be logged or captured. As per , operators must be aware that such data will be at risk of compromise. As such, measures should be taken to firstly reduce the risk of compromise and secondly reduce the risk of abuse of compromised data. While a full discussion of both aspects is out of scope for this document, the following paragraphs discuss high-level considerations that can be applied to qlog data. To reduce the risk of compromise, operators can take measures such as: limiting the length of time that data is stored, encrypting data in transit and at rest, limiting access rights to the data, and auditing data usage practices. qlog deployments that provide integrated options for automated or manual data deletion and (aggressive) aggregation, improve the ability to minimize the risk of compromise. To reduce the risk of data abuse after compromise, data can be anonymized, pseudonymized, otherwise permutated/replaced, truncated, (re-)encrypted, or aggregated. A partial discussion of applicable techniques (especially for IP address information) can be found in . Operators should, however, be aware that many of these techniques have been shown to be insufficient to safeguard user privacy and/or to protect user identity, especially if a qlog data set is large or easily correlated against other data sources. Finally, qlog operators should consider the interplay between their use case needs and end user rights or preferences. While active user participation (as indicated by ) on a per-qlog basis is difficult, as logs are often captured out-of-band to the main user interaction and intent, general user expectations should be taken into account. qlog deployments that provide mechanisms to integrate the capture, storage and removal of qlogs with more general, often pre-existing, user preference and privacy control systems, improve the ability to protect data sensitive or confidential to the end user. In qlog, these data are typically (but not exclusively) contained in fields of the RawInfo type (see ). qlog users should thus be particularly hesitant to include these fields for all but the most stringent use cases.

IANA Considerations TODO: primarily the .well-known URI

References Normative References JavaScript Object Notation (JSON) is a lightweight, text-based, language-independent data interchange format. It was derived from the ECMAScript Programming Language Standard. JSON defines a small set of formatting rules for the portable representation of structured data. This document removes inconsistencies with other specifications of JSON, repairs specification errors, and offers experience-based interoperability guidance. I-JSON (short for "Internet JSON") is a restricted profile of JSON designed to maximize interoperability and increase confidence that software can process it successfully with predictable results. This document describes the JavaScript Object Notation (JSON) text sequence format and associated media type "application/json-seq". A JSON text sequence consists of any number of JSON texts, all encoded in UTF-8, each prefixed by an ASCII Record Separator (0x1E), and each ending with an ASCII Line Feed character (0x0A). In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document proposes a notational convention to express Concise Binary Object Representation (CBOR) data structures (RFC 7049). Its main goal is to provide an easy and unambiguous way to express structures for protocol messages and data formats that use CBOR or JSON. A content media type name sometimes includes partitioned meta- information distinguished by a structured syntax to permit noting an attribute of the media as a suffix to the name. This document defines several structured syntax suffixes for use with media type registrations. In particular, it defines and registers the "+json", "+ber", "+der", "+fastinfoset", "+wbxml" and "+zip" structured syntax suffixes, and provides a media type structured syntax suffix registration form for the "+xml" structured syntax suffix. This document is not an Internet Standards Track specification; it is published for informational purposes. JavaScript Object Notation (JSON) is a lightweight, text-based, language-independent data interchange format. It was derived from the ECMAScript Programming Language Standard. JSON defines a small set of formatting rules for the portable representation of structured data. This document removes inconsistencies with other specifications of JSON, repairs specification errors, and offers experience-based interoperability guidance. This document describes the JavaScript Object Notation (JSON) text sequence format and associated media type "application/json-seq". A JSON text sequence consists of any number of JSON texts, all encoded in UTF-8, each prefixed by an ASCII Record Separator (0x1E), and each ending with an ASCII Line Feed character (0x0A). Structured syntax suffixes for media types allow other media types to build on them and make it explicit that they are built on an existing media type as their foundation. This specification defines and registers "+json-seq" as a structured syntax suffix for JSON text sequences. This RFC documents the format used for Comma-Separated Values (CSV) files and registers the associated MIME type "text/csv". This memo provides information for the Internet community. This specification defines a lossless compressed data format that is compatible with the widely used GZIP utility. This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. This specification defines a lossless compressed data format that compresses data using a combination of the LZ77 algorithm and Huffman coding, with efficiency comparable to the best currently available general-purpose compression methods. The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. These design goals make it different from earlier binary serializations such as ASN.1 and MessagePack. This document specifies version 1.3 of the Transport Layer Security (TLS) protocol. TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery. This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961. This document also specifies new requirements for TLS 1.2 implementations. This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. This document offers guidance for developing privacy considerations for inclusion in protocol specifications. It aims to make designers, implementers, and users of Internet protocols aware of privacy-related design choices. It suggests that whether any individual RFC warrants a specific privacy considerations section will depend on the document's content. This document presents operational, policy, and security considerations for DNS recursive resolver operators who choose to offer DNS privacy services. With these recommendations, the operator can make deliberate decisions regarding which services to provide, as well as understanding how those decisions and the alternatives impact the privacy of users. This document also presents a non-normative framework to assist writers of a Recursive operator Privacy Statement, analogous to DNS Security Extensions (DNSSEC) Policies and DNSSEC Practice Statements described in RFC 6841. Informative References KU Leuven Facebook Protocol Labs This document describes concrete qlog event definitions and their metadata for QUIC events. These events can then be embedded in the higher level schema defined in [QLOG-MAIN]. KU Leuven Facebook Protocol Labs This document describes concrete qlog event definitions and their metadata for HTTP/3 and QPACK-related events. These events can then be embedded in the higher level schema defined in [QLOG-MAIN]. This document describes the Concise Binary Object Representation (CBOR) Sequence format and associated media type "application/cbor-seq". A CBOR Sequence consists of any number of encoded CBOR data items, simply concatenated in sequence. Structured syntax suffixes for media types allow other media types to build on them and make it explicit that they are built on an existing media type as their foundation. This specification defines and registers "+cbor-seq" as a structured syntax suffix for CBOR Sequences. This document defines the wire image, an abstraction of the information available to an on-path non-participant in a networking protocol. This abstraction is intended to shed light on the implications that increased encryption has for network functions that use the wire image.

Change Log

Since draft-ietf-quic-qlog-main-schema-03:

• Added security and privacy considerations discussion (#252)

Since draft-ietf-quic-qlog-main-schema-02:

• No changes - new draft to prevent expiration

Since draft-ietf-quic-qlog-main-schema-01:

• Change the data definition language from TypeScript to CDDL (#143)

Since draft-ietf-quic-qlog-main-schema-00:

• Changed the streaming serialization format from NDJSON to JSON Text Sequences (#172)
• Added Media Type definitions for various qlog formats (#158)
• Changed to semantic versioning

Since draft-marx-qlog-main-schema-draft-02:

• These changes were done in preparation of the adoption of the drafts by the QUIC working group (#137)
• Moved RawInfo, Importance, Generic events and Simulation events to this document.
• Added basic event definition guidelines
• Made protocol_type an array instead of a string (#146)

Since draft-marx-qlog-main-schema-01:

• Decoupled qlog from the JSON format and described a mapping instead (#89)
  • Data types are now specified in this document and proper definitions for fields were added in this format
  • 64-bit numbers can now be either strings or numbers, with a preference for numbers (#10)
  • binary blobs are now logged as lowercase hex strings (#39, #36)
  • added guidance to add length-specifiers for binary blobs (#102)
• Removed "time_units" from Configuration. All times are now in ms instead (#95)
• Removed the "event_fields" setup for a more straightforward JSON format (#101,#89)
• Added a streaming option using the NDJSON format (#109,#2,#106)
• Described optional optimization options for implementers (#30)
• Added QLOGDIR and QLOGFILE environment variables, clarified the .well-known URL usage (#26,#33,#51)
• Overall tightened up the text and added more examples

Since draft-marx-qlog-main-schema-00:

• All field names are now lowercase (e.g., category instead of CATEGORY)
• Triggers are now properties on the "data" field value, instead of separate field types (#23)
• group_ids in common_fields is now just also group_id

Acknowledgements Much of the initial work by Robin Marx was done at the Hasselt and KU Leuven Universities. Thanks to Jana Iyengar, Brian Trammell, Dmitri Tikhonov, Stephen Petrides, Jari Arkko, Marcus Ihlar, Victor Vasiliev, Mirja Kuehlewind, and Jeremy Laine for their feedback and suggestions.