Proxy for Congestion Notification

describes a congestion notification message called Fast Congestion Notification Packet (CNP), which can be sent by a congested network node to the traffic sender directly. Fast CNP extends the CNP consumed by the traffic sender supporting Remote Direct Memory Access (RDMA) over Converged Ethernet version 2 (RoCEv2). RoCEv2 runs the InfiniBand transport layer over UDP and IP protocols on an Ethernet network, bringing many of the advantages of InfiniBand to Ethernet networks. For a traffic sender supporting RoCEv2, congestion control is important, and the RoCEv2 CNP or RoCEv2 Fast CNP must be used to alert the sender slowing down the transmission rate. For a traffic sender not supporting RoCEv2, congestion control is still important, and the corresponding congestion notification mechanism supported by the sender must be used to alert the sender slowing down the transmission rate. Considering there are multiple different congestion notification mechanisms existing for the traffic sender, if a congested network node would send a congestion notification message to the traffic sender directly, it's not easy for the congested network node to know what kind of congestion notification mechanism is supported by the traffic sender. In the case that the congested network node doesn't know exactly what kind of congestion notification mechanism is supported by each specific traffic sender within the network, a proxy network node nearer to the specific traffic sender can help to do the transition as a translator. That is to say, the congested network node sends a congestion notification message to a proxy network node first, and then the proxy network node notifies the traffic sender about the congestion, as long as the proxy network node knows what kind of congestion notification mechanism is supported by the traffic sender. This document describes the necessity and feasibility to introduce a proxy network node between the congested network node and the traffic sender. Specifically, the problem statement is described in Sections 1 and 3, and the format of the congestion notification message sent from the congested network node to the proxy network node is defined in Section 4, and the potential solutions on how the congested network node knows the address of the proxy node are defined in Section 5.

ABR: Area Border Router CNP: Congestion Notification Packet DoS: Denial-of-Service ECN: Explicit Congestion Notification ELC: Entropy Label Capability ELCv3: Entropy Label Characteristic IBTA: InfiniBand Trade Association PFC: Priority-based Flow Control PNC: Proxy Node Capability RDMA: Remote Direct Memory Access RoCEv2: RDMA over Converged Ethernet version 2

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

In the field of congestion control, there are at least four kinds of referenced congestion notification mechanisms. This document introduces the fifth congestion notification mechanism called fast congestion notification with proxy. The first congestion notification mechanism is referred to as classical stepwise back pressure with dedicated Ethernet pause frame, as shown in Figure 1.

|Network|<===>|Network|<===>|Network|<===>|Traffic | |Sender | |Node 1 | |Node 2 | |Node 3 | |Receiver| +--------+ +-------+ +-------+ +-------+ +--------+ Congestion Point ]]> With this congestion notification mechanism, the congested network node (Netwok Node 3 in Figure 1) notifies the directly connected upstream network node (Network Node 2 in Figure 1) about the congestion by a dedicated Ethernet pause frame called Priority-based Flow Control (PFC) frame, and then the upstream network node may stepwise notify its directly connected upstream network node about the congestion by a PFC frame, until the most upstream network node (Network Node 1 in Figure 1) may notify the directly connected traffic sender about the congestion by a PFC frame. details how this kind of congestion notification mechanism works. The second congestion notification mechanism is referred to as classical congestion notification without dedicated packet, as shown in Figure 2.

| +--------+ +-------+ +-------+ +-------+ +--------+ |Traffic |<===>|Network|<===>|Network|<===>|Network|<===>|Traffic | |Sender | |Node 1 | |Node 2 | |Node 3 | |Receiver| +--------+ +-------+ +-------+ +-------+ +--------+ Congestion Point ]]> With this congestion notification mechanism, the traffic sender indicates that it supports the congestion notification from the traffic receiver by a specific Explicit Congestion Notification (ECN) marking within the IP header of the data packet, and the congested network node (Netwok Node 3 in Figure 2) notifies the traffic receiver about the congestion by a specific ECN marking. After receiving a data packet with the specific ECN marking, the traffic receiver would notify congestion to the traffic sender by a specific TCP marking within the TCP header of the data packet. details how this kind of congestion notification mechanism works. The third congestion notification mechanism is referred to as classical congestion notification with dedicated packet, as shown in Figure 3.

| +--------+ +-------+ +-------+ +-------+ +--------+ |Traffic |<===>|Network|<===>|Network|<===>|Network|<===>|Traffic | |Sender | |Node 1 | |Node 2 | |Node 3 | |Receiver| +--------+ +-------+ +-------+ +-------+ +--------+ Congestion Point ]]> With this congestion notification mechanism, the traffic sender indicates that it supports the congestion notification from the traffic receiver by a specific ECN marking within the IP header of the data packet, and the congested network node (Netwok Node 3 in Figure 3) notifies the traffic receiver about the congestion by a specific ECN marking. After receiving a data packet with the specific ECN marking, the traffic receiver would notify congestion to the traffic sender by a dedicated congestion notification packet. details an example on how this kind of congestion notification mechanism works. The fourth congestion notification mechanism is referred to as fast congestion notification without proxy, as shown in Figure 4.

|Network|<===>|Network|<===>|Network|<===>|Traffic | |Sender | |Node 1 | |Node 2 | |Node 3 | |Receiver| +--------+ +-------+ +-------+ +-------+ +--------+ Congestion Congestion Notification Point Proxy ]]> With this congestion notification mechanism, the congested network node (Netwok Node 3 in Figure 5) notifies the proxy network node about the congestion by a dedicated congestion notification packet, and then the proxy network node notifies the traffic sender about the congestion by a congestion notification mechanism supported by the traffic sender. This document details how this kind of congestion notification mechanism works, except that the congestion notification mechanism between the proxy network node and the traffic sender is beyond the scope of this document.

The congestion notification message sent from the congested network node to the proxy network node is a UDP message which is formatted as follows:

UDP Header: The UDP header as specified in includes the UDP source port, UDP destination port, UDP length, and UDP checksum. A well-known UDP destination port needs to be allocated for this Congestion Notification Message. IP Five-Tuple: The IP five-tuple as described in includes the source IP address, destination IP address, protocol number, source port number, and destination port number. The IP five-tuple is copied from the data packet causing congestion, and it's used to identify a flow for which the transmission rate needs to be reduced by the traffic sender. Congestion Level: This 3-bit field indicates the congestion level. Value 0 of this field represents the lowest congestion level and value 7 of this field represents the highest congestion level.

Before the congested network node can send the congestion notification message to the proxy network node, the congested network node has to be notified about the IP address of the proxy network node. There are two kinds of notification mechanisms. One kind of notification mechanism is through control plane advertisement, which is specified in Section 5.1. Another kind of notification mechanism is through data plane dissemination, which is specified in Section 5.2.

Even though Proxy Node Capability (PNC) is a property of the node, in some cases it is advantageous to associate and advertise the PNC with a prefix. When PNC is advertised with a prefix, that means the congested network node should send the congestion notification packet to the proxy network node but not the traffic sender associated with that prefix.

Analogous to the Entropy Label Capability (ELC) Flag (E-flag) defined in Section 3 of , a new bit PNC Flag (P-flag) is defined, which is Bit 7 in the Prefix Attribute Flags , as shown in Figure 7.

P-Flag: PNC Flag (Bit 7) Set for the local host prefix of the originating node if it's used as a congestion notification proxy node for the prefix. The PNC signaling MUST be preserved when a router propagates a prefix between ISIS levels .

Analogous to the ELC Flag (E-flag) defined in Section 3.1 of , a new bit PNC Flag (P-flag) is defined, which is Bit 2 in OSPFv2 Prefix Attribute Flags field , as shown in Figure 8.

P-Flag: PNC Flag (Bit 2) Set for the local host prefix of the originating node if it's used as a congestion notification proxy node for the prefix. The PNC signaling MUST be preserved when an OSPFv2 Area Border Router (ABR) distributes information between areas. To do so, an ABR MUST originate an OSPFv2 Extended Prefix Opaque LSA including the received PNC setting.

Analogous to the ELC Flag (E-flag) defined in Section 3.2 of , a new bit PNC Flag (P-flag) is defined, which is Bit 2 in OSPFv3 Prefix Attribute Flags field , as shown in Figure 9.

P-Flag: PNC Flag (Bit 2) Set for the local host prefix of the originating node if it's used as a congestion notification proxy node for the prefix. The PNC signaling MUST be preserved when an OSPFv3 Area Border Router (ABR) distributes information between areas. The setting of the PNC Flag in the Inter-Area-Prefix-LSA or in the Inter-Area-Prefix TLV , generated by an ABR, MUST be the same as the value the PNC Flag associated with the prefix in the source area.

Analogous to the Entropy Label Characteristic (ELCv3) TLV defined in Section 3.1 of , a new PNC characteristic TLV is defined, which uses code value TBD2 in "BGP Next Hop Dependent Characteristic Codes" registry requested by , as shown in Figure 10.

PNC TLV: code TBD2, length 0, and carries no value Carried for the local host prefix of the originating node if it's used as a congestion notification proxy node for the prefix.

This section specifies an extension to the data packet when sent from the traffic sender. Specifically, an IPv6 Destination Options header with one IPv6 destination option carrying the IPv6 address of the proxy network node is added to the data packet. The congested network node would copy the IPv6 address of the proxy network node from the IPv6 destination option of data packet causing congestion to the destination address field of the congestion notification packet. The Proxy Network Node Address Destination Option has the following format:

Option Type: This 8-bit field identifies the type of Option that needs to be allocated. defines how to encode the three high-order bits of the Option Type field. The two high-order bits specify the action that must be taken if the processing IPv6 node does not recognize the Option Type; for this Option, these two bits MUST be set to 00 (skip over this option and continue processing the header). The third-highest-order bit specifies whether the Option Data can change en route to the packet's final destination; for this Option, the value of this bit MUST be set to 0 (Option Data does not change en route). Opt Data Len: This 8-bit field indicates the length of the Option Data field of this Option in bytes. This field MUST be set to 16. Option Data: This 16-octet field indicates the proxy network node's IPv6 address to which the congested network node would send the congestion notification message defined in Section 4.

The congestion notification from congested network node to the proxy network node MUST be applied in a specific controlled domain. A limited administrative domain provides the network administrator with the means to select, monitor, and control the access to the network, making it a trusted domain. To avoid potential Denial-of-Service (DoS) attacks, it is RECOMMENDED that implementations apply rate-limiting policies when generating and receiving congestion notification messages. A deployment MUST ensure that border-filtering drops inbound congestion notification message from outside of the domain and that drops outbound congestion notification message leaving the domain. A deployment MUST support the configuration option to enable or disable the congestion notification proxy feature defined in this document. By default, the congestion notification proxy feature MUST be disabled. As this document describes new option for IPv6, the security considerations of and apply.

This document requests the following allocations from IANA: - A well-known UDP port number TBD1 in the "Service Name and Transport Protocol Port Number" registry is requested to be assigned to the Congestion Notification Message. - Bit 7 in the "IS-IS Bit Values for Prefix Attribute Flags Sub-TLV" registry is requested to be assigned to the PNC Flag (P-Flag). - Bit 2 in the "OSPFv2 Prefix Attribute Flags" registry is requested to be assigned to the PNC Flag (P-Flag). - Bit 2 in the "OSPFv3 Prefix Attribute Flags" registry is requested to be assigned to the PNC Flag (P-Flag). - Code value TBD2 in the "BGP Next Hop Dependent Characteristic Codes" registry is requested to be assigned to the PNC characteristic TLV. - An IPv6 Option Type in the "Destination Options and Hop-by-Hop Options" registry is requested to be assigned to the Proxy Node Address Destination Option.

The author would like to acknowledge Jinghai Yu and Shaofu Peng for the very helpful discussion.