Fast congestion notification for distributed RoCEv2 network based on SRv6

The scenarios mentioned in Chapter 3 can be summarized as the network diagram shown in Figure 1. In these scenarios, the sender in DC needs to transmit RoCEv2 packets to the receiver in another DC through SRv6 tunnels in WAN, the process is as follows: The sender in DC sends RoCEv2 packets to WAN's ingress PE through the leaf, spine, and gateway devices. At the WAN's ingress PE, the RoCEv2 packets are encapsulated according to the SRv6 tunnel protocol. The WAN's P node transits the payload from ingress PE to egress PE through SRv6 tunnels. At the WAN's egress PE, the payload are decapsulated to RoCEv2 packets and transmitted to the receiver in DC.

Within DC, RoCEv2 implements congestion control based on PFC and ECN. In WAN, it is necessary to extend PFC and ECN so that fast notifications can be achieved when congestion is detected by WAN devices.

PFC is the feature used in ethernet to prevent data loss due to congestion. In PFC, traffic is classified into 8 priorities according to the 802.1Q protocol, and each priority maintains its queue. When the queue length of the router's receive port exceeds a certain threshold, the router sends a PAUSE frame to upstream to stop transmitting traffic. When the queue length of the router's receive port drops below a certain threshold, the router sends a RESUME frame to upstream to continue transmitting traffic. As WAN devices often carry multiple services simultaneously, if PFC is triggered due to the congestion in a specific service, it may impact other services on the port and pose security risks. Therefore, when carrying RoCEv2 packets through tunnel in WAN, precise flow control needs to be implemented based on traffic information such as inner IP header, outer IP header and priority. This document proposes several parameters for the ARN mechanism ([I-D. draft-wh-rtgwg-adaptive-routing-arn]), enabling fast notification for congested flow in WAN.

[I-D. draft-wh-rtgwg-adaptive-routing-arn] propose the ARN packet format as follows, as well as two parameters: the Para-Type Bit 0 for flow identifier and the Para-Type Bit 1 for path identifier. Specifically, flow identifier is based on the five-tuple from packet header to indicate the affected flow, path identifier is based on the 32-bit path ID to uniquely identify the affected path in the network.

This document further defines the Para-Type Bit 2 for priority identifier and the Para-Type Bit 3 for compression time.

When bit2 of Para-Type is 1, the following parameter is concluded in Parameters to indicate the priority of affected flows:

When bit3 of Para-Type is 1, the following parameter is concluded in Parameters to indicate the compression time of affected flows:

Taking Figure 1 as an example, within DC1 and DC2, PFC is still used to implement the flow control based on port priority. Within WAN, precise flow control can be used to achieve congestion control at three levels (service level, priority level, and flow level), effectively avoiding impact on other services. Specifically, when the receiving port of device detects congestion, it will notify upstream devices to pause the corresponding flow: When configured to stop the service in case of congestion, the router will pass on ARN packet carrying flow identifier (identify the corresponding service based on the five-tuple of outer IP header) and compression time to upstream devices. When configured to stop specific flow in case of congestion, the router will pass on ARN packet carrying flow identifier (identify the corresponding flow based on the five-tuple of inner IP header) and compression time to upstream devices. When configured to stop flow based on priority, the router will pass on ARN packet carrying priority identifier and compression time to upstream devices. Upon receiving the ARN packet, upstream devices will halt the corresponding flow based on the configured mode. Between WAN and DC1 or DC2, considering the capabilities of the gateway devices, congestion control can be achieved by PFC or precise flow control.

ECN enables a forwarding element (e.g., a router) to notify the sender for congestion control without having to drop packets [RFC3168]. When a router detects congestion, instead of dropping packets as a signal of congestion, it marks the packets with an ECN codepoint in the IP header. The marked packets alert the receiver that packet loss is imminent, and then the receiver alerts the sender by sending Congestion Notification Packet (CNP). After receiving the CNP, the sender knows to slow down the transmission rate temporarily until the flow path is ready to handle a higher rate of traffic. Within WAN, devices should be able to notify the sender to reduce the transmission rate when congestion is detected. [RFC6040] redefines how the explicit congestion notification (ECN) field of the IP header should be constructed on entry to and exit from any IP-in-IP tunnel (including the SRv6 tunnel). It defines the rules for ingress encapsulation and egress decapsulation as follows:

(!!!)| | ECT(0) | ECT(0) | ECT(0) | ECT(1) | CE | | ECT(1) | ECT(1) | ECT(1) (!) | ECT(1) | CE | | CE | CE | CE | CE(!!!)| CE | +---------+---------+------------+------------+------------+ Currently unused combinations are indicated by '(!!!)' or '(!)' Figure 6: Tunnel egress decapsulation behaviors]]> At the ingress PE, there are two encapsulation modes: a REQUIRED 'normal mode' and a 'compatibility mode', which is for backward compatibility with tunnel egress do not understand ECN. In normal mode, the ingress PE constructs the outer encapsulating IP header by copying the two-bit ECN field of the incoming IP header. In compatibility mode, it clears the ECN field in the outer header to the Not-ECT codepoint. At the egress PE, to decapsulate the inner header at the tunnel egress, The ECN field in the outgoing header is set to the codepoint at the intersection of the appropriate arriving inner header (row) and arriving outer header (column) in Figure 4, or the packet is dropped where indicated. To enable congestion control for WAN devices, the ingress PE MUST use the normal mode to construct the outer encapsulating IP header. When a WAN device detects congestion (e.g. R2), it sets the ECN field of the outer IP header to CE. Then, the egress PE sets the ECN field in the outgoing IP header to CE during decapsulation. When receiver receives the marked packet, it sends CNP warning to the sender to slow down the transmission rate of the corresponding flow, thereby preventing congestion.

[RFC7514] defines Really Explicit Congestion Notification (RECN), also known as Fast Congestion Notification Packet (Fast CNP). By extending the RoCEv2 CNP, Fast CNP can be sent by the intermediate router directly to the sender, advising the sender to reduce the transmission rate at which it sends the flow of RoCEv2 data traffic. When transporting RoCEv2 packets through SRv6 tunnels in WAN, intermediate devices are unable to send Fast CNP back to the ingress PE based on the information in RoCEv2 packets. Additionally, the ingress PE is unable to recognize the Fast CNP and therefore cannot forward them to the sender. Therefore, this document proposes a Congestion SID (END.CON) to address aforementioned issues. END.CON is a 128-bit value configured on PE and disseminated to other routers via IGP. The endpoint action associated with END.CON is to decapsulate the packet and then look up the routing table for traffic forwarding. The workflow of END.CON is as follows: When the ingress PE passes RoCEv2 packets through SRv6 tunnels, END.CON is encapsulated as the source address of outer IP header. When WAN devices detect congestion, they encapsulate an outer IP header (destination address is set to END.CON) outside the Fast CNP and send it back to the ingress PE. When the ingress PE detects that the destination address of the received packet hits END.CON, it will decapsulate the Fast CNP and pass it to the sender in DC.