Proactive Flow Control Point Detection in WAN

Introduction With the growth of intelligent computing services, scenarios such as disaggregated computing and real-time inference require the lossless transmission of large volumes of bursty traffic over wide-area networks (WANs). To achieve lossless data transmission over WAN, there're quite a few recent works aiming to deploy flow control mechanisms in WAN to avoid packet loss in case of congestion, e.g, discusses how to deploy PFC(Priority-based Flow Control, ) in WAN based on SRv6 data plane, and proposes the method of precise/fine-grained flow control to achieve flow control at the network slice level. To conclude, to deploy flow control mechanism in WAN, the node facing congestion needs to generate a flow control message and sends it to the upstream point which is able to perform the flow control action (e.g, stop sending the corresponding traffic or reducing the sending rate). The flow control message sending mechanism may include one of the follows:

Multicast. Although standard PFC propagates congestion information via Ethernet multicast frames, multicast-based mechanism is not preferred in WAN since it cannot accurately reach upstream nodes, potentially leading to incorrect flow suppression and impacting unrelated services.
Centralized configuration. The controller, with the awareness of all the node and the path information in the network, can configure the information of the upstream flow control point on each node that may generate the flow control message. But this methods will bring extra burden for the controller in large scale networks.
Distributed decision. The congested node decides the upstream node itself. The difficulty is that the congested node needs to be aware of the necessary information to make the proper decision.

Based on the above considerations, this document proposes a proactive detection mechanism for flow control in WAN, letting the congested node to know precisely which upstream point should the flow control message be sent to. The detailed flow control mechanism itself is out of the scope of this document.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Flow Control in WAN

+----+ --> +----+ --> +----+ --> +----+ | R1 | | R2 | | R3 | | R4 | | R5 | +----+ +----+ +----+ +----+ +----+ ]]> As shown in Figure 1, data center A and data center B are connected via a path(R1-R2-R3-R4-R5) in WAN. R1,R2,R4 and R5 are able to perform the function of flow control, and R3 is a legacy device which doesn't support any flow control technology. When congestion occurs at R4, R4 generates a flow control message (e.g, the PFC pause frame defined in , the congestion notification message defined in ) to the nearest upstream stream node that is able to perform flow control, i.e, node R2. R2 receives the notification and performs the flow control based on the content of the notification message and local capacity. If R2 cannot handle the congestion, a flow control message is forwarded further upstream to R1.

Proactive Flow Control Point Detection Method The basic concept of the proactive flow control point detection method in this document is to send a flow control detection packet along the packet forwarding path. When receiving the flow control detection packet, the node that is capable of flow control updates the packet with its own information(e.g, the interface address or the corresponding SRv6 ajacency SID of the interface), so the detection packet will always carry the information of the nearest upstream node that's capable of flow control and the node receiving the detection packet would store this information and use it as the destination of the flow control message when congestion occurs.

Packet Format The following information is required in the flow control detection packet:

Upstream flow control point identifier: indicate the nearest upstream point(interface of the node) that is able to perform flow control for the corresponding traffic flow
Flow control object identifier: used in the scenario of precise flow control to provide the extra information of flow control object, e.g, if the flow control is at the network slice level, the network slice ID is the flow control object identifier
Path identifier: used to identify the path of the traffic flow when necessary.

A new Hop-by-Hop option (Section 4.3 of ) type is defined in this document to carry the fields above for flow control point detection. Its format is shown in Figure 2.

Option Type: 8-bit identifier of the type of option. The type of Flow Control Detection option is TBA. Opt Data Len: 8-bit unsigned integer indicates the length of the option Data field of this option, in octets. Flags: 8-bit flags field. The most significant bit is defined in this document.

P (PFC): The P flag is used to indicate whether this flow control detection packet is used for PFC.

Upstream Type: indicates the type of the upstream point that is able perform flow control for the traffic. When set to 1, the upstream Point Info carries a 128-bit IPv6 interface address, when set to 2, the upstream Point Info carries a 128-bit SRv6 adj-SID. Upstream Point Info: 128-bit field carrying the corresponding upstream point information based on the value of the Upstream Type. Context Type: The Context Type field is an 8-bit bitmap that specifies which contexts are included in the Context field of the packet. Each bit in this field corresponds to a specific context. When a bit is set to 1, it indicates the presence of the corresponding parameter, where,

bit 0: indicates the presence of the path identifier field when set, the format of the path identifier field is shown as in section 4.1.1
bit 1: indicates the presence of the flow control object identifier when set,the format of the flow control object is shown as in section 4.1.2.

The packet fields defined above can be carried in-band or out-band as long as the packet is forwarded along the same path of the normal traffic flow

bit 0 Context When bit0 of Context Type is 1, the following context is included to indicate the identifier of the path: Path ID: used to identify a path in the network. The scope of the Path ID is implementation specific.

bit 1 Context When bit1 of Context Type is 1, the following context is included to carry the identifier of the flow control object when precise flow control mechanism is used: O-Type: 8-bit field, indicates the type of Flow Control Object ID. When the value of the O-Type is 1, it indicates that Flow Control Object ID carries a 32-bit network slice ID.

Processing Procedures As in Figure 1, the traffic of network slice 1 is forwarded along an SRv6 path, the corresponding SID list is <SID-R12,SID-R23,SID-R45>, whereas SID-R12 is the adjacency SID of R1 for the adjacency between R1 and R2, SID-R23 and SID-R45 are also the corresponding adj-SID on R2 and R4. And precise flow control is enabled in the network to control the congestion at the network slice level. R1,R2,R4 and R5 are able to perform flow control at the network slice level, and R3 is a legacy device which doesn't support any flow control technology. To detect the flow control point along the path of slice 1, R1 sends a flow control detection packet in band with the traffic of slice 1, since R1 is capable of flow control, R1 puts SID-R12 into the Upstream Point Info and the Flow Control Object ID field is set to slice-ID 1. When R2 receives the packet, it stores the mapping between slice 1 and SID-R12, and updates the Flow Control Object ID with its own information, i.e, SID-R23. Since R3 doesn't recognize the flow control detection packet, it just forwards the packet based on the SID-list and slice-ID of the packet. When R4 receives the packet, it stores the mapping between slice 1 and SID-R23, and updates the Flow Control Object ID with its own information, i.e, SID-R45. When R5 receives the packet, it stores the mapping between slice 1 and SID-R45, and since R5 is the endpoint, it stops processing the packet further. When congestion occurs at R4 in slice 1, R4 would generate a flow control message for slice 1, and based on the local information, R4 finds the information of upstream flow control point, i.e, SID-R23, and uses it as the destination of the flow control message. When R2 receives the flow control message with DA set as local adj-SID SID-R23, R2 perform the flow control for slice 1 on the port related with SID-R23 based on the context of the flow control message. If R2 is not able to control the congestion and generates a flow control message further, R2 would send the message with DA set to SID-R12 based on the local information.

Security Considerations The security considerations with IPv6 Hop-by-Hop Options header are described in ,, , , . This document introduces a new IPv6 Hop-by-Hop option which is either processed in the fast path or ignored by network nodes, thus it does not introduce additional security issues.

IANA Considerations This document requests IANA to assign a new option type from "Destination Options and Hop-by-Hop Options" registry .