Requirements from Control and Management Viewpoint for Collective Communication Optimization

In recent years, with the development and evolution of various applications and business, especially the rapid growth of AI applications, distributed computing performance has become more and more important and has gradually become a key factor restricting the growth of these applications. As the primary communication mode of current distributed computing systems, the performance of collective communication is crucial. However, there exists many problems to be solved for collective communication to improve performance. On the one hand, many collective communication operations implemented by message-level communication libraries like MPI and NCCL mainly depend on the unicast point-to-point communication mechanism, leading to the redundancy of network information, the underutilization of network resources and the waste of network capabilities. On the other hand, since the underlying network protocols and collective communication are not co-designed, there is a semantic gap between inter-process message transportation and packet forwarding. Therefore, there is huge space for the optimization of collective communication. At present, the industry and academia are also actively promoting the development, implementation and deployment of collective communication optimization topic. The research group Computing in the Network, COIN for short, also focus on this topic. Their work goal is mainly to investigate how network data plane programmability can improve Internet architecture, with a too broad focus scope including network functions offloading, machine learning acceleration, in-network caching and in-network control, etc. In addition to the solution of collective operation offloading COIN talk about for collective communication optimization, multicast substituting for single point unicast, scheduling tasks and planning transportation paths by topology awareness and bridge semantic gap between inter-process message transportation and packet forwarding can also play the role of optimizing collective communication. This draft provide some necessary requirements from the network control and management viewpoint, combined with the optimization solutions of collective communication offloading, multicast mechanisms, topology awareness and semantic gap bridge between inter-process message transportation and packet forwarding, to guideline the standardization work of collective communication optimization.

Scarce memory resources provided by network devices for collective communication MUST be scheduled and controlled, e.g. assigning a scheduling priority to collective communication offloading tasks. Compared to the amount of collective communication message in the applications such as AI, Big Data System, HPC, etc., it is severely mismatched and extremely scarce for memory resource provided by network devices for collective communication, such as network programmable switches. Use Case. The memory of programmable switch is scarce for the amount of gradient transmitted in distributed training. There is some existing work to solve this problem like pool-based streaming and dynamic sharing, which are not enough yet. A use case of fully utilizing the memory of programmable switch is that the control and management module of switch assigns a priority to the aggregation task, to dynamically and preemptively schedule the aggregation tasks in the data plane, thus making more full use of memory in the form of switch aggregators.

Topology awareness and mapping work are REQUIRED to be done to put some of the end-host computing on the network nodes for collective communication optimization. In many collective operation tasks, the logical relationship between nodes is usually described in the form of graph, and then mapping to the physical network. Therefore, collective communication offloading requires awareness of the network topology and making efficient mappings. Use Case. In the parameter server architecture commonly used in distributed training, the parameter server can be reasonably mapped to spine switches in the fat tree physical network with being aware of network topology. Under this mapping mechanism, the traffic path is more simplified and the traffic volume of the whole network is greatly compressed. Compared to the traditional collective communication mode, the optimized end-to-network or end-to-network-to-end one with topology awareness and mapping makes the physical topology and the logical topology closer, more friendly and unified.

Logical Topology +----------------+ |Parameter Server| +--------+-------+ | +----------+---+------------+ | | | +----+---+ +----+---+ +---+----+ |Worker 1| |Worker 2| ...... |Worker n| +--------+ +--------+ +--------+ |Mapping | +----------+---------+ |Management & Control| | | | Topology Awareness | | Paths Planning | +----------+---------+ | |Mapping v Physical Topology +-----+ +-----+ |Spine| |Spine| +--+--+ +--+--+ | | +----------+-+------------++-----------+ | | | | +--+--+ +--+--+ +--+--+ +--+--+ |Leaf | |Leaf | |Leaf | |Leaf | +--+--+ +--+--+ +--+--+ +--+--+ | | | | +--+--+ +--+--+ +--+--+ +--+--+ | | | | | | | | +-+-+ +-+-+ +-+-+ +-+-+ +-+-+ +-+-+ +-+-+ +-+-+ |GPU| |GPU| |GPU| |GPU| |GPU| |GPU| |GPU| |GPU| +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+

Some collective communication interfaces MUST be defined and managed for application developers to shield tedious network engineering details, such as flow control, packet organization, chip-specific programming language, etc. If not, applications developers will need too much arcane knowledge and expertise, which is beyond their willingness and prevent from the evolution of the emerging applications. Use case. The industry and academy have actually proposed some abstractions of collective communication operations, such as collective communication libraries MPI, NCCL, NetRPC, etc. In the control plane, these interfaces need to be configured and instantiated to complete the part of collective communication functionality.