Proxy for Congestion Notification

describes a congestion notification message called Fast Congestion Notification Packet (Fast CNP), which can be sent by a congested network node to the traffic sender directly. Fast CNP extends the CNP consumed by the traffic sender supporting Remote Direct Memory Access (RDMA) over Converged Ethernet version 2 (RoCEv2). RoCEv2 has already been widely deployed, and it runs the InfiniBand transport layer over UDP and IP protocols on an Ethernet network, bringing many of the advantages of InfiniBand to Ethernet networks. For a traffic sender supporting RoCEv2, congestion control is important, and the RoCEv2 CNP or RoCEv2 Fast CNP must be used to alert the sender slowing down the transmission rate. For a traffic sender not supporting RoCEv2, congestion control is still important, and the corresponding congestion notification message supported by the sender must be used to alert the sender slowing down the transmission rate. Considering there are multiple different congestion notification messages existing for the traffic sender, if a congested network node would send a congestion notification message to the traffic sender directly, there is a prerequisite for the congested network node to know what kind of congestion notification message is supported by each specific traffic sender; Secondly, when the congested network node is a VPN Provider (P) router, it's difficult for the congested network node to send a congestion notification message to the traffic sender directly, because there are different routing domains for the VPN P router and VPN Customer Edge (CE) router; Thirdly, when the traffic sender supports RoCEv2, it's difficult for the congested network node to construct a standard RoCEv2 CNP (please refer to Section 3 of ). A proxy network node between the congested network node and the traffic sender can help to resolve the problems described above, being independent of the extension proposed in . The congested network node sends a congestion notification message to a proxy network node first, and then the proxy network node notifies the traffic sender about the congestion using a congestion notification message known by the traffic sender (e.g., the standard RoCEv2 CNP). For the selection of the proxy network node, there are at least three rules. First one, the selected proxy network node must know what kind of congestion notification message is supported by the traffic sender; Second one, the selected proxy network node and the congested network node must be within the same routing domain; Third one, for RoCEv2 network, the selected proxy network node must be able to learn the mapping table between Source Queue Pair and Destination Queue Pair through data traffic, which means the selected proxy network node must be located where both the forward direction traffic and the backward direction traffic need to traverse. How to select a proxy network node for a specific traffic sender is deployment specific and beyond the scope of this document. This document describes the necessity and feasibility to introduce a proxy network node between the congested network node and the traffic sender. Specifically, the problem statement is described in Sections 1 and 3, and the format of the congestion notification message sent from the congested network node to the proxy network node is defined in Section 4, and the solution on how the congested network node knows the address of the proxy node is defined in Section 5.

ABR: Area Border Router CNP: Congestion Notification Packet DoS: Denial-of-Service ECN: Explicit Congestion Notification ELC: Entropy Label Capability ELCv3: Entropy Label Characteristic IBTA: InfiniBand Trade Association PNC: Proxy Node Capability RDMA: Remote Direct Memory Access RoCEv2: RDMA over Converged Ethernet version 2

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

In the field of congestion control, there are at least three kinds of referenced congestion notification mechanisms. This document introduces the fourth congestion notification mechanism called "Fast Congestion Notification with Proxy". The first congestion notification mechanism is referred to as classical congestion notification without dedicated packet, as shown in .

| +--------+ +-------+ +-------+ +-------+ +--------+ |Traffic |<===>|Network|<===>|Network|<===>|Network|<===>|Traffic | |Sender | |Node 1 | |Node 2 | |Node 3 | |Receiver| +--------+ +-------+ +-------+ +-------+ +--------+ Congestion Point ]]> With this congestion notification mechanism, the traffic sender indicates that it supports the congestion notification from the traffic receiver by a specific Explicit Congestion Notification (ECN) marking within the IP header of the data packet, and the congested network node (Netwok Node 3 in ) notifies the traffic receiver about the congestion by a specific ECN marking. After receiving a data packet with the specific ECN marking, the traffic receiver would notify congestion to the traffic sender by a specific TCP marking within the TCP header of the data packet. details how this kind of congestion notification mechanism works. The second congestion notification mechanism is referred to as classical congestion notification with dedicated packet, as shown in .

| +--------+ +-------+ +-------+ +-------+ +--------+ |Traffic |<===>|Network|<===>|Network|<===>|Network|<===>|Traffic | |Sender | |Node 1 | |Node 2 | |Node 3 | |Receiver| +--------+ +-------+ +-------+ +-------+ +--------+ Congestion Point ]]> With this congestion notification mechanism, the traffic sender indicates that it supports the congestion notification from the traffic receiver by a specific ECN marking within the IP header of the data packet, and the congested network node (Netwok Node 3 in ) notifies the traffic receiver about the congestion by a specific ECN marking. After receiving a data packet with the specific ECN marking, the traffic receiver would notify congestion to the traffic sender by a dedicated congestion notification packet. details an example on how this kind of congestion notification mechanism works. The third congestion notification mechanism is referred to as fast congestion notification without proxy, as shown in .

|Network|<===>|Network|<===>|Network|<===>|Traffic | |Sender | |Node 1 | |Node 2 | |Node 3 | |Receiver| +--------+ +-------+ +-------+ +-------+ +--------+ Congestion Point ]]> With this congestion notification mechanism, the congested network node (Netwok Node 3 in ) notifies the traffic sender about the congestion directly by a dedicated congestion notification packet. details an example on how this kind of congestion notification mechanism works. The fourth congestion notification mechanism is referred to as fast congestion notification with proxy, as shown in .

|Network|<===>|Network|<===>|Network|<===>|Traffic | |Sender | |Node 1 | |Node 2 | |Node 3 | |Receiver| +--------+ +-------+ +-------+ +-------+ +--------+ Congestion Congestion Notification Point Proxy ]]> With this congestion notification mechanism, the congested network node (Netwok Node 3 in ) notifies the proxy network node about the congestion by a dedicated congestion notification packet, and then the proxy network node notifies the traffic sender about the congestion by a congestion notification message supported by the traffic sender. This document details how this kind of congestion notification mechanism works, except that the specific congestion notification message between the proxy network node and the traffic sender is beyond the scope of this document.

The congestion notification message sent from the congested network node to the proxy network node can be a UDP message or an ICMP message, if a UDP message it's formatted as follows:

UDP Header: The UDP header as specified in includes the UDP source port, UDP destination port, UDP length, and UDP checksum. A well-known UDP destination port needs to be allocated for this Congestion Notification Message. IP Five-Tuple: The IP five-tuple as described in includes the source IP address, destination IP address, protocol number, source port number, and destination port number. The IP five-tuple is copied from the data packet causing congestion, and it's used to identify a flow for which the transmission rate needs to be reduced by the traffic sender. When the congested network node is a VPN P router, the IP five-tuple is carried below the VPN encapsulation. Congestion Level: This 3-bit field indicates the congestion level. Value 0 of this field represents the lowest congestion level and value 7 of this field represents the highest congestion level.

Before the congested network node can send the congestion notification message to the proxy network node, the congested network node has to know about the IP address of the proxy network node. The proxy network node can notify the congested network node of its IP address by advertising its proxy capability in advance. Even though the Proxy Node Capability (PNC) is a property of the node, in some cases it is advantageous to associate and advertise the PNC with a prefix. When PNC is advertised with a prefix, that means the congested network node should send the congestion notification packet to the proxy network node but not the traffic sender associated with that prefix.

Analogous to the Entropy Label Capability (ELC) Flag (E-flag) defined in Section 3 of , a new bit PNC Flag (P-flag) is defined, which is Bit 7 in the Prefix Attribute Flags , as shown in .

P-Flag: PNC Flag (Bit 7) Set for the local host prefix of the originating node if it's used as a congestion notification proxy node for the prefix. The PNC signaling MUST be preserved when a router propagates a prefix between ISIS levels .

Analogous to the ELC Flag (E-flag) defined in Section 3.1 of , a new bit PNC Flag (P-flag) is defined, which is Bit 2 in OSPFv2 Prefix Attribute Flags field , as shown in .

P-Flag: PNC Flag (Bit 2) Set for the local host prefix of the originating node if it's used as a congestion notification proxy node for the prefix. The PNC signaling MUST be preserved when an OSPFv2 Area Border Router (ABR) distributes information between areas. To do so, an ABR MUST originate an OSPFv2 Extended Prefix Opaque LSA including the received PNC setting.

Analogous to the ELC Flag (E-flag) defined in Section 3.2 of , a new bit PNC Flag (P-flag) is defined, which is Bit 2 in OSPFv3 Prefix Attribute Flags field , as shown in .

P-Flag: PNC Flag (Bit 2) Set for the local host prefix of the originating node if it's used as a congestion notification proxy node for the prefix. The PNC signaling MUST be preserved when an OSPFv3 Area Border Router (ABR) distributes information between areas. The setting of the PNC Flag in the Inter-Area-Prefix-LSA or in the Inter-Area-Prefix TLV , generated by an ABR, MUST be the same as the value the PNC Flag associated with the prefix in the source area.

Analogous to the Entropy Label Characteristic (ELCv3) TLV defined in Section 3.1 of , a new PNC characteristic TLV is defined, which uses code value TBD2 in "BGP Next Hop Dependent Characteristic Codes" registry requested by , as shown in .

PNC TLV: code TBD2, length 0, and carries no value Carried for the local host prefix of the originating node if it's used as a congestion notification proxy node for the prefix.

The congestion notification from congested network node to the proxy network node MUST be applied in a specific controlled domain. A limited administrative domain provides the network administrator with the means to select, monitor, and control the access to the network, making it a trusted domain. To avoid potential Denial-of-Service (DoS) attacks, it is RECOMMENDED that implementations apply rate-limiting policies when generating and receiving congestion notification messages. A deployment MUST ensure that border-filtering drops inbound congestion notification message from outside of the domain and that drops outbound congestion notification message leaving the domain. A deployment MUST support the configuration option to enable or disable the congestion notification proxy feature defined in this document. By default, the congestion notification proxy feature MUST be disabled.

This document requests the following allocations from IANA: - A well-known UDP port number TBD1 in the "Service Name and Transport Protocol Port Number" registry is requested to be assigned to the Congestion Notification Message. - Bit 7 in the "IS-IS Bit Values for Prefix Attribute Flags Sub-TLV" registry is requested to be assigned to the PNC Flag (P-Flag). - Bit 2 in the "OSPFv2 Prefix Attribute Flags" registry is requested to be assigned to the PNC Flag (P-Flag). - Bit 2 in the "OSPFv3 Prefix Attribute Flags" registry is requested to be assigned to the PNC Flag (P-Flag). - Code value TBD2 in the "BGP Next Hop Dependent Characteristic Codes" registry is requested to be assigned to the PNC characteristic TLV.

The author would like to acknowledge Jinghai Yu and Shaofu Peng for the very helpful discussion.