Notification for Adaptive Routing

Introduction Large-scale supercomputing centers require interconnection of large-scale computing nodes. However, the scaling-out of clusters increases network latency and deployment costs, which cannot meet computing power and deployment requirements. Directly connected network topology (such as Dragonfly) shows the advantages of scalability with small network diameter, which is widely adopted in HPC and supercomputing systems networks. In the network that adopts the directly connected topology, there are multiple but non-equivalent paths to the destination node. In most cases, the shortest path is preferred to be selected for forwarding traffic. However, traffic congestion or link failures may occur on the shortest path. To this end, adaptive routing is widely used for nodes to make dynamic routing decisions based on dynamics of network topology (e.g., link failure) as well as variations of traffic (e.g., link congestion). By proactively detecting link congestion status, the network node could forward packets along a shorter but non-congested path, improving overall throughput and resilience as well as reducing the latency. When the link is non-congested, packets are forwarded over the shortest path. When congestion occurs on the shortest path, the local node that detects it applies adaptive routing immediately and, at the same time, explicitly advertises congestion signals to other remote nodes.
In this way, the network selects another non-congested but non-shortest path to forward packets temporarily until congestion elimination signal is received. Adaptive routing enables the network to mitigate traffic collisions and make use of idle links to improve bandwidth utilization. This document proposes a proactive congestion notification mechanism for adaptive routing, and describes the conditions when to trigger the dissemination, as well as what information to carry in ARN. Adaptive Routing Notifications (ARNs) are not only applicable to directly connected topologies such as Dragonfly, but to any topologies that aim to apply dynamic multipath optimization. ARN is also useful for advertising failures of link or interface, in which case traffic is desired to bypass the failed path.

Terminology AR: Adaptive Routing ARN: Adaptive Routing Notification BPT: Best Path Table

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

ARN Mechanism ARN can be triggered whenever local congestion is detected to appear or disappear. Congestion signal is sent by the detected node to other nodes of interests.

Topology Example depicts a simplified dragonfly topology (only relevant links are drawn). The nodes in each Group are directly connected to each other. The groups are all connected with direct links. As shown in , Node1 has a direct link connecting Group1 and Group2. When the direct link (Node1 <-> Group2) is congested, all nodes of Group1 should be notified and immediately update the path selection policy. For example, partial or all flows originating from group1 to group2 may choose Group3 as transmit instead of using direct link (Node1 <-> Group2) until congestion elimination.

Triggering ARN The local node could determine whether congestion occurs by monitoring interface status, such as bandwidth utilization and queue depth of the interface. When the monitored value exceeds the preset threshold, the state is determined to be in congestion and congestion notification is triggered. When the monitored value falls back below the preset threshold, the state is determined to be in non-congestion and a notification of congestion elimination is triggered. When the local node detects any change in congestion status, it can send the corresponding ARN continuously to other network nodes in the same group. The notifications can be sent to multiple nodes using multicast technology provided by the network. ARN packets SHOULD be set as high priority to ensure that they can be processed in a timely manner. The congestion level is RECOMMENDED to be present in ARN in order for fine-grained control of adaptive routing.

ARN for Congestion Detection An ARN packet for congestion detection SHOULD include the Severity information which is used to indicate the level of congestion or the type of failure. Whenever a network node receives an ARN packet indicating congestion detection, if the optimal forwarding path in the local best path table (BPT) should pass through the relevant interface, the network node deletes the path from the BPT and choose other sub-optimal paths. How to organize and maintain BPT is out of scope in this document. An ARN packet for congestion detection MUST include neccesary information (e.g., ID of peer group connected by the compromised link) to locate susceptible paths in BPT.

ARN for Congestion Elimination When the network node receives the ARN that represents congestion elimination, it checks that whether the Cost value of the forwarding path through the relevant interface (P1) is less than the forwarding path stored in the current BPT (P2), the forwarding path (P1) is stored in the BPT and replaces the current path (P1) in the table. How to organize and maintain BPT is out of scope in this document. An ARN packet for congestion elimination MUST include neccesary information (e.g., ID of peer group connected by the compromised link) to locate susceptible paths in BPT.

Security Considerations TBD.

IANA Considerations TBD.