]> Google

martin.h.duke@gmail.com

Web and Internet Transport Transport and Services Working Group udp ecn Explicit Congestion Notification (ECN) applies to all transport protocols in principle. However, it had limited deployment for UDP until QUIC became widely adopted. As a result, documentation of UDP socket APIs for ECN on various platforms is sparse. This document records the results of experimenting with these APIs in order to get ECN working on UDP for Chromium on Apple, Linux, and Windows platforms. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Transport and Services Working Group Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction reserves two bits in the IP header for Explicit Congestion Notification (ECN), which provides network feedback to endpoint congestion controllers. This has historically mostly been relevant to TCP (), where any incoming ECN codepoints are internally consumed by the kernel, and therefore imply no application interface except enabling and disabling the capability. The Stream Control Transport Protocol (SCTP) () has long supported ECN in its design. SCTP is sometimes carried over DTLS and UDP (). In principle, user-space implementers might have leveraged UDP ECN APIs to deliver ECN codepoints between SCTP and the UDP socket. The author is not aware of any such efforts. defines ECN over RTP over UDP. The author is aware of a research implementation, but cannot confirm any commercial deployments. However, QUIC runs over UDP and has seen wider deployment than SCTP. The Low Latency, Low Loss, Scalable Throughput (L4S) experiment () and QUIC have combined to increase interest in ECN over UDP. The Chromium Projects () provide a widely-deployed protocol library that includes QUIC. An effort to provide ECN support for QUIC on the many platforms on which Chromium is deployed revealed that many ECN-related UDP socket interfaces are poorly documented. This informational document provides a record of that experience, to encourage further support for ECN in other QUIC implementations, and indeed any consumer of ECN codepoints that operates over UDP. It is not a standards-track document and does not bind platforms to any API, or suggest any such API. Many socket APIs continue to reference the "ToS (Type of Service) byte" even though obsoleted that 25 years ago. That 8-bit field now contains a 6-bit Differentiated Services Code Point (DSCP), in addition to the ECN bits. This document focuses on the APIs for the C and C++ languages. Other languages are likely to have different syntax and capabilities.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document is not a general tutorial on UDP socket programming, and assumes familiarity with basic socket concepts like binding, socket options, and common system error codes.

Receiving ECN codepoints Network devices can change the ECN codepoint in the IP header. Since this feedback is required at the packet sender, the packet receiver needs to extract this codepoint from the UDP socket in order to report to the sender. There are two components to this: setting the socket to report incoming ECN marks, and retrieving the value for each incoming packet.

Setting the socket to report incoming ECN codepoints

Linux, Apple and FreeBSD To report ECN, applications set a socket option to true using a setsockopt() call. IPv6 sockets require a socket option of level IPPROTO_IPV6 and name IPV6_RECVTCLASS. IPv4 sockets require a socket option of level IPPROTO_IP and name IP_RECVTOS. For dual-stack sockets, on Linux hosts the application sets both the IPV6_RECVTCLASS and IP_RECVTOS options to receive ECN codepoints on all incoming packets. On Apple and FreeBSD hosts, the application only sets the IPPROTO_IPV6-level socket option with name IPV6_RECVTCLASS to receive codepoints for both v4 and v6; setting an IPPROTO_IP-level socket option on an IPv6 socket results in an error. In particular this applies to the IPPROTO_IP-level socket option with the name IP_RECVTOS. At the time of writing, an example implementation can be found at .

Windows Windows documentation recommends using the function WSASetRecvIPEcn() to enable ECN reporting regardless of the IP version. This function dates to Windows 10 Build 20348, according to . However, this can also be accomplished by calling setsockopt() and using options of level IPPROTO_IP and name IP_RECVECN for IPv4, and IPPROTO_IPV6 and IPV6_RECVECN for IPv6. These options are documented at . For dual-stack sockets, WSASetRecvIPEcn() will not enable ECN reporting for IPv4. This requires a separate setsockopt() call using the IP_RECVECN option. If a socket is bound to a IPv6-mapped IPv4 address (i.e. it is of the format ::ffff:<IPv4 address>), calls to WSASetRecvIpEcn() return error EINVAL. These sockets should instead use an explicit setsockopt() call to set IP_RECVECN. At the time of writing, an example implementation can be found at .

Retrieving ECN codepoints on incoming packets All platforms described in this document require the use of a recvmsg() call to read data from the socket to retrieve ECN information, because that information is encoded in the control data that is returned from that function. Those platforms all return zero or more "cmsg" that contain requested information about the arriving packet. Examples of the technique described below can be found at and .

Linux If the incoming packet is IPv4, Linux will include a cmsg of level IPPROTO_IP and type IP_TOS. The cmsg data contains an unsigned char. If the incoming packet is IPv6, Linux will include a cmsg of level IPPROTO_IPV6 and type IPV6_TCLASS. The cmsg data contains an int. The resulting report contains the entire IP header byte, which includes other fields. The ECN codepoint constitutes the two least-significant bits of this byte. The same applies to the Linux-specific recvmmsg() call.

Apple and FreeBSD If a UDP message (UDP/IPv4) is received on an IPv4 socket, the ancillary data will contain a cmsg of level IPPROTO_IP and type IP_RECVTOS. The cmsg data contains an unsigned char. If a UDP message (UDP/IPv6 or UDP/IPv4) is received on an IPv6 socket, the ancillary data will contain a cmsg of level IPPROTO_IPV6 and type IPV6_TCLASS. The cmsg data contains an int. The provided data is the entire byte from the IP header, which includes other fields. The ECN codepoint constitutes the two least-significant bits of this byte.

Windows If the incoming packet is IPv4, the socket will include a cmsg of level IPPROTO_IP and type IP_ECN. The cmsg data contains an int. If the incoming packet is IPv6, the socket will include a cmsg of level IPPROTO_IPV6 and type IPV6_ECN. The cmsg data contains an int. The resulting integer solely consists of the ECN codepoint, and requires no further bitwise operations.

Sending ECN codepoints Existing ECN specifications envision a particular connection consistently sending the same ECN codepoint. It might transition that marking after successfully completing a handshake, recognizing the path or the peer do not support ECN, or transitioning to a new path. Therefore, using a socket option to configure a consistent marking is generally more resource-efficient. However, some server designs receive all incoming packets on a single socket. As the many connections that constitute this packet stream may have different support for ECN, it is suitable to configure outgoing ECN on a per-packet basis.

On a per-socket basis

Linux and Apple Both Linux and Apple platforms set the outgoing ECN for IPv4 packets with a socket option of level IPPROTO_IP and name IP_TOS. For IPv6 packets, they use level IPPROTO_IPV6 and name IPV6_TCLASS. This setsockopt() call also sets the Differentiated Services Code Point (DSCP) bits that make up the rest of the header byte. Applications making this call will generally want to preserve any existing DSCP setting, which might require a getsockopt() call. For dual-stack sockets, Linux requires an additional setsockopt() call with IP_TOS. Apple sockets do not and will return an error if this call is made. An example of the technique described above can be found at .

Windows At the time of this writing, Windows does not provide a way to configure marking on a per-socket basis.

On a per-packet basis Packets can be individually marked with ECN codepoints using the control information that accompanies a sendmsg() call.

Linux and Apple These platforms expect a cmsg with level IPPROTO_IP and type IP_TOS if the destination is an IPv4 address, or a IPv4-mapped IPv6 address. Otherwise, they expect a cmsg with level IPPROTO_IPV6 and type IPV6_TCLASS. The same applies to the Linux-specific sendmmsg() call.

Microsoft Windows uses a cmsg with level IPPROTO_IP and type IP_ECN for IPv4 packets. Windows uses a cmsg with level IPPROTO_IPV6 and type IPV6_ECN for IPv6 packets. An example of the technique described above can be found at .

Security Considerations The security implications of ECN are documented in and . This document is a guide to enabling these capabilities, which incurs no additional security considerations. Note that implementing ECN capabilities on some platforms, but not others, can help peers identify the operating system in use by a host, which can have privacy implications. This document aims to mitigate that possibility.

IANA Considerations This document has no IANA actions.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References n.d. n.d. n.d. n.d. n.d. This memo specifies the incorporation of ECN (Explicit Congestion Notification) to TCP and IP, including ECN's use of two bits in the IP header. [STANDARDS-TRACK] This document specifies the Transmission Control Protocol (TCP). TCP is an important transport-layer protocol in the Internet protocol stack, and it has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFCs 1011 and 1122, and it should be considered as a replacement for the portions of those documents dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. This document describes the Stream Control Transmission Protocol (SCTP) and obsoletes RFC 4960. It incorporates the specification of the chunk flags registry from RFC 6096 and the specification of the I bit of DATA chunks from RFC 7053. Therefore, RFCs 6096 and 7053 are also obsoleted by this document. In addition, RFCs 4460 and 8540, which describe errata for SCTP, are obsoleted by this document. SCTP was originally designed to transport Public Switched Telephone Network (PSTN) signaling messages over IP networks. It is also suited to be used for other applications, for example, WebRTC. SCTP is a reliable transport protocol operating on top of a connectionless packet network, such as IP. It offers the following services to its users: The design of SCTP includes appropriate congestion avoidance behavior and resistance to flooding and masquerade attacks. The Stream Control Transmission Protocol (SCTP) is a transport protocol originally defined to run on top of the network protocols IPv4 or IPv6. This document specifies how SCTP can be used on top of the Datagram Transport Layer Security (DTLS) protocol. Using the encapsulation method described in this document, SCTP is unaware of the protocols being used below DTLS; hence, explicit IP addresses cannot be used in the SCTP control chunks. As a consequence, the SCTP associations carried over DTLS can only be single-homed. This memo specifies how Explicit Congestion Notification (ECN) can be used with the Real-time Transport Protocol (RTP) running over UDP, using the RTP Control Protocol (RTCP) as a feedback mechanism. It defines a new RTCP Extended Report (XR) block for periodic ECN feedback, a new RTCP transport feedback message for timely reporting of congestion events, and a Session Traversal Utilities for NAT (STUN) extension used in the optional initialisation method using Interactive Connectivity Establishment (ICE). Signalling and procedures for negotiation of capabilities and initialisation methods are also defined. [STANDARDS-TRACK] This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. This document describes the L4S architecture, which enables Internet applications to achieve low queuing latency, low congestion loss, and scalable throughput control. L4S is based on the insight that the root cause of queuing delay is in the capacity-seeking congestion controllers of senders, not in the queue itself. With the L4S architecture, all Internet applications could (but do not have to) transition away from congestion control algorithms that cause substantial queuing delay and instead adopt a new class of congestion controls that can seek capacity with very little queuing. These are aided by a modified form of Explicit Congestion Notification (ECN) from the network. With this new architecture, applications can have both low latency and high throughput. The architecture primarily concerns incremental deployment. It defines mechanisms that allow the new class of L4S congestion controls to coexist with 'Classic' congestion controls in a shared network. The aim is for L4S latency and throughput to be usually much better (and rarely worse) while typically not impacting Classic performance. This document defines the IP header field, called the DS (for differentiated services) field. [STANDARDS-TRACK]

Acknowledgments The author would like to thank Ryan Hamilton, who provided constant advice through this effort. Randall Meyer from Apple and Nick Grifka from Microsoft provided useful hints about the behavior of their respective operating systems. However, the author takes full responsibility for any errors above. Neal Cardwell, Gorry Fairhurst, Max Franke, Rodney Grimes, Will Hawkins, Guillaume Hétier, Max Inden, Jonathan Lennox, Colin Perkins, Marten Seemann, and Michael Tuexen made improvements to this draft.