DetNet Data Plane Enhancements for Large-Scale Deterministic Networks

According to , Deterministic Networking (DetNet) operates at the IP layer and delivers service which provides extremely low data loss rates and bounded latency within a network domain. The framework of DetNet data planes has been specified in RFC8938. The IP and MPLS DetNet data plane has been defined respectively in RFC8939 and RFC8964. The DetNet IP data plane primarily uses 6-tuple-based flow identification. And the DetNet MPLS data plane leverages existing pseudowire (PW) encapsulations and MPLS Traffic Engineering (MPLS-TE) encapsulations. The applications in 5G networks demand much more deterministic and precise properties in large-scale networks. The existing deterministic technologies are facing large-scale number of nodes and long-distance transmission, traffic scheduling, dynamic flows, and other controversial issues in large-scale networks. The enhanced DetNet Data plane is required to support a data plane method of flow identification and packet treatment. has described the enhancement requirements for DetNet data plane. The enhanced DetNet data plane aims to describe how to use IP and/or MPLS, and related OAM, to support a data plane method of flow identification and packet treatment over Layer 3. The enhanced QoS-related functions and metadata should be provided in large-scale networks. For example, as described in , the end-to-end bounded latency depends on the value of queuing delay bound along with the queuing mechanisms. Mutiple queuingmechanisms can be used to guarantee the bounded latency in DetNet. New DetNet-specific metadata should be carried in enhanced DetNet IP/MPLS/SRv6 Data Plane. This document analyzes the gaps of the existing technologies especially applying the DetNet data plane as per RFC8938 and proposes the enhancement of packet treatment to support the functions and metadata for enhanced DetNet data plane. It describes related enhanced controller plane considerations as well.

The terminology is defined as and .

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

5G network is oriented to the internet of everything. It need to supports the Ultra-reliable Low Latency Communications (uRLLC) services. The uRLLC services demand SLA guarantees such as low latency and high reliability and other deterministic and precise properties especially in Wide Area Network (WAN) applications.The uRLLC services should be provided in large-scale networks which cover the industries such as intelligent electrical network, intelligent factory, internet of vehicles, industry automation and other industrial internet scenarios. The industrial internet is the key infrastructure that coordinate various units of work over various system components, e.g. people, machines and things in the industrial environment including big data, cloud computing, Internet of Things (IOT), Augment Reality (AR), industrial robots, Artificial Intelligence (AI) and other basic technologies. For the intelligent electrical network, there are deterministic requirements for communication delay, jitter and packet loss rate. For example, in the electrical current difference model, a delay of 3~10ms and a jitter variation is no more than 100us are required. For the automation control, it is one of the basic application and the the core is closed-loop control system. The control process cycle is as low as millisecond level, so the system communication delay needs to reach millisecond level or even lower to ensure the realization of precise control. There are three levels of real-time requirements for industrial interconnection: factory level is about 1s, and process level is 10~100ms, and the highest real-time requirement is motion control, which requires less than 1ms. So the deterministic latency requirements are different with varying services and network scenarios. As defined in , the DetNet QoS can be expressed in terms of : Minimum and maximum end-to-end latency, bounded jitter (packet delay variation), packet loss ratio and an upper bound on out-of-order packet delivery. As described in , DetNet applications differ in their network topologies and specific desired behavior and different services requires differentiated DetNet QoS. In the large-scale networks, multiple services with differentiated DetNet QoS is co-existed in the same DetNet network. The classification of the deterministic flows within different levels is should be taken into considerations. It is required to provide Latency, bounded jitter and packet loss dynamically and flexibly in all scenarios for each characterized flow. As the Figure 1 shows, the services can be divided into 5 levels and level 2~5 is the DetNet flows and level-1 is non-DetNet flow. DetNet applications and DetNet QoS is differentiated within each level.

+-------------+-----------+----------+----------+----------+-----------+ | Item | Level-1 | Level-2 | Level-3 | Level-4 | Level-5 | +-------------+-----------+----------+----------+----------+-----------+ | Applications| Broadcast | Voice | Audio and| AR/VR | Industrial| | Examples | | | Video | | | +-------------+-----------+----------+----------+----------+-----------+ | DetNet QoS | Bandwidth | Jitter | Latency | Low | Ultra-low | | | Guarantee | Guarantee| Guarantee| latency |latency and| | | | | |and jitter| jitter | +-------------+-----------+----------+----------+----------+-----------+ From the perspective of deterministic service requirements, deterministic Quality of Service (QoS) in the network can be divided into five types or levels: Level-1: bandwidth guarantee. The indicator requirements include basic bandwidth guarantee and certain packet loss tolerance. There is no requirement for the upper bound of the latency, and no requirement for the jitter. Typical services include download and FTP services. Level-2: jitter guarantee. The indicator requirements include: jitter 50ms, delay 300ms. Typical services include synchronous voice services, such as voice call. Level-3: Latency guarantee. The indicator requirements include: delay 50ms, jitter 50ms. Typical services include real-time communication services, such as video, production monitoring, and communication services. Level-4: low delay and low jitter guarantee. The indicator requirements include: delay 20ms, jitter 5ms. Typical services include video interaction services, such as AR/VR, holographic communication, cloud video and cloud games. Level-5: ultra-low delay and jitter guarantee. The indicator requirements include: delay 10ms, jitter 100us. Typical services include production control services, such as power protection and remote control. Moreover, different DetNet services is required to tolerate different percentage of packet loss ratio such as 99.9%, 99.99%, 99.999%, and so on.

Traditional Ethernet, IP and MPLS networks which is based on statistical multiplexing provides best-effort packet service and offers no delivery and SLA guarantee. As described in , the primary technique by which DetNet achieves its QoS is to allocate sufficient resources. But it can not be achieved by not sufficient resource which can be allocated due to practical and cost reason. So it is required to achieve the high-efficiency of resources utilization when provide the DetNet service.

As described in , deterministic forwarding can only apply to flows with such well-defined characteristics as periodicity and burstiness. As defined in DetNet architecture , the traffic characteristics of an App-flow can be CBR (constant bit rate) or VBR (variable bit rate) of L1, L2 and L3 layers (VBR takes the maximum value when reserving resources). But the current scenarios and technical solutions only consider CBR flow, without considering the coexistence of VBR and CBR, the burst and aperiodicity of flows. The operations such as shaping or scheduling have not been specified. Even TSN mechanisms are based on a constant and forecastable traffic characteristics. It will be more complicated in a large-scale network where much more flows coexist and the traffic characteristics is more dynamic. A huge number of flows with different DetNet QoS requirements is dynamically concurrent and the state of each flow cannot be maintained. It is required to offer reliable delivery and SLA guarantee for dynamic flows. For example, periodic flow and aperiodic flow (including micro burst flow, etc.), CBR and VBR flow, flow with different periods or phases, etc. When the network needs to forward these deterministic flows at the same time, it must solve the problems of time micro bursts, queue processing and aggregation of multiple flows.

In large-scale applications, the network topology may consists of a large number of nodes and links which leads to difficulty with controlling the end-to-end delay and jitter. High speed, long-distance transmission and asymmetric links may also co-exists and affects the bounded latency such as increasing transmission latency, jitter and packet loss in large-scale networks. The network topology in a large-scale network may across multiple domains within a single administrative control or a closed group of administrative control as per . Moreover, DetNet domains or nodes may be interconnected with different sub-network technologies such as FlexE tunnels, TSN sub-network, IP/MPLS/SRv6 tunnels and so on. It is required to support the inter-domain deterministic metric and routes to achieve the end-to-end bounded latency.

As defined in , the DetNet data plane describes how application flows, or App-flows are carried over DetNet networks and it is provided by the DetNet service and forwarding sub-layers with DetNet-related data plane functions and mechanisms. This section analyzes the DetNet technical gaps when applying the DetNet data plane as per RFC8938 in large-scale networks.

In , the DetNet data plane can provide the DetNet-Specific Metadata such as Flow-ID for both the service and forwarding sub-layers. The flow-based state information is required to be maintained for per-flow processing rules. For example, the resource reservation configuration is required for each flow. DetNet as per provides the capability to aggregate the individual flows to downscale the operations of flow states. However, it still requires large amount of control signaling to establish and maintain DetNet flows. It may be challenging for network operations with a large number of deterministic flows and network nodes in large-scale networks.

As described in , the primary goals are to achieve the DetNet QoS to provide minimum and maximum end-to-end latency and bounded jitter, low packet loss ratio and an upper bound on out-of-order packet delivery. But the data plane particularly focuses on the DetNet service sub-layer which provides a set of Packet Replication, Elimination, and Ordering Functions (PREOF) functions to provide end-to-end service assurance. It mainly provides the capabilities for DetNet to guarantee the reliability. The DetNet forwarding sub-layer provides corresponding forwarding assurance with IETF existing functions using resource allocations and explicit routes. But these functions can not provide the deterministic latency (bounded latency, low packet loss and in-order delivery) assurance in large-scale networks. The following sections mainly discuss the gap analysis for the forwarding sub-layer functions to provide deterministic latency assurance.

Traditional routes only have reachability. As per , explicit optimized paths with allocation of resources should be provided to achieve the DetNet QoS. But the deterministic requirements such as end-to-end delay and jitter are only used as path computation constraints. Multiple network metrics which are measured and distributed by the routing system should be taken into consideration. In large-scale networks, it may be challenging to compute the best path to meet all of the requirements. In multi-domain scenarios, the inter-domain deterministic routes need to be established and provisioned. Especially when interconnecting with sub-networks, the selection of intra-domain paths acrossing cooperating domains should consider the bounded latency in each domain and the stitching of the paths. Moreover, the paths vary with the real-time change of the network topology. On the basic of the resources, the steering path and routes for deterministic flows should be programmed before the flows coming and able to provide SLA capability. And the routes should be considered to be established in distributed and centralized control Plane. As described in , the packet replication and elimination service protection should be provided to achieve the low packet loss ratio. It will copy the flows and spread the data over multiple disjoint forwarding paths. The bounded latency and jitter of each path should be meet service deterministic requirement. And the difference of latency within these paths should be limited. So the replication and elimination deterministic routes with configured latency and jitter policy should be taken into consideration. It is required to generate two disjoint paths with very close delay to form 1+1 protection and perform concurrent transmission and dual reception, and make replication and elimination on the egress PE.

As per , the forwarding sub-layer uses buffer resources for packet queuing, as well as reservation and allocation of bandwidth capacity resources. In large-scale networks, the bandwidth, buffer and scheduling resources are combined with queuing mechanisms to guarantee the deterministic latency. The reservation and allocation of queuing related resources or deterministic latency resources should be taken into consideration in DetNet data plane.

As per , the forwarding sub-layer provides the QoS-related functions needed by the DetNet flow including the use of queuing techniques. But the queuing techniques which are defined in existing IETF technologies can not guarantee the bounded latency such as Active Queue Management(AQM). And the queuing mechanisms which are defined in IEEE802.1 TSN can not be directly applied in large-scale networks such Time Aware Shaping [IIEEE802.1Qbv] and Cyclic Queuing and Forwarding [IEEE802.1Qch] with time synchronization. Enhancement of queuing mechanisms have been discussed in DetNet such as cyclic-scheduling queuing mechanism , deadline-scheduling queuing mechanism and , and asynchronous queuing mechanism . The function of multiple queuing mechanisms and related DetNet-Specific Metadata has not been defined in DetNet data plane.

As defined in , the DetNet data plane describes how application flows, or App-flows are carried over DetNet networks and it is provided by the DetNet service and forwarding sub-layers with DetNet-related data plane functions and mechanisms. From charter and milestones, the enhanced DetNet data plane is required to provide the enhancemant of flow identification and packet treatment including the enhanced QoS-related functions and metadata in large-scale networks.

This section proposes the enhancement for the DetNet Data Plane Protocol Stack as shown in Figure 1 and the enhanced DetNet-related data plane functions and mechanisms should be provided by the DetNet service and forwarding sub-layers.

From the perspective of differentiated services requirements in section 3.1.1, a large-scale network needs to provide the deterministic service for various applications. And the deterministic service may demand different DetNet QoS levels according to different application scenarios. The DetNet data plane should support the identification of multiple flows and the differentiated deterministic QoS for each DetNet flow. According to the gap described in section 3.3.1, this document proposes the enhanced DetNet data plane to support flow identification of DetNet differentiated services with service-level identification. It may downscale the network operations with a large number of deterministic flows and network nodes in large-scale networks.

As discussed in section 3.3.2.1, it may be challenging to compute the best path to meet all of the requirements and the the paths vary with the real-time change of the network topology in large-scale networks. The explicit routes may be not appropriate for large-scale networks. This document propose the deterministic routes which can be strict explicit paths or loose routes. The former is applicable to centralized scenarios with controllers, and the latter is applicable to distributed scenarios.

As discussed in section 3.3.2.1, it may be challenging to compute the best path to meet all of the requirements within a large-scale network topology pool including multiple network metrics. This document proposes the deterministic links to provide a one-dimensional deterministic metric to guarantee for the deterministic forwarding capabilities at different levels. The computing end-to-end delay bounds is defined in . It is the sum of non-queuing delay bound and queuing delay bound in DetNet bounded latency model. The upper bounds of queuing delay depends on the queuing mechanisms deployed along the path. For example, a link with a queuing mechanism that does not guarantee a bounded delay a non-determinisitc link and a link with a queuing mechanism that can provide deterministic delay is called a deterministic link. The delay of a a deterministic link is consist of the propagation delay of the packet on the link and the queuing delay of the packet at the node. A deterministic link can be a sub-network that provides deterministic transmission or a Point-to-Point (P2P) link. The deterministic links could be distributed by IGP protocol as per .

As discussed in section 3.3.2.1, the inter-domain deterministic routes need to be established and provisioned in multi-domain scenarios. The stitching of the intra-domain paths should be considered in DetNet data plane. In the centralized scenario, when the source and destination PEs of a deterministic service are located at the two ends with a limited physical range, one controller (single domain) or multiple controllers (cross domains) compute one or more paths with deterministic SLA according to the typical Traffic Specification (T-SPEC) based on the collected deterministic resources, or compute dynamically according to the service T-SPEC as required by the services. In the distributed scenario, deterministic loose routes are computed on the device through routing protocols. Interior Gateway Protocol (IGP) is used to compute deterministic routes based on deterministic-delay inside a domain, and Border Gateway Protocol (BGP) is used to compute deterministic routes based on accurate delay/jitter across domains.

As discussed in section 3.3.2.2, the reservation and allocation of queuing related resources or deterministic latency resources should be taken into consideration in DetNet data plane. The networks need to shield the differences between network capabilities. Deterministic resource is the basis for providing deterministic network services. It refers to the resources that meet the deterministic indicators of a node and link processing as well as the corresponding resource processing mechanisms (such as link bandwidth, queues, and scheduling algorithms). It is required to make unified modeling for all the deterministic resources. The deterministic links are provided and distributed to support the deterministic resource and forwarding capabilities. As discussed in section 3.1.2, it is necessary to make overall resource planning and scheduling for the network to achieve the high-efficiency of resources utilization when provide multiple DetNet services. The admission control policy of a flow should take into account the deterministic resource.

As dicussed in section 3.3.2.3, it is required to support the enhancement of queuing mechanisms. Multiple queuingmechanisms can provide different levels of latency, jitter and other guarantees. The DetNet forwarding sub-layer may provide the function and technology such as multiple queuing and traffic treatment for DetNet application flows. The DetNet data plane may also encode the queuing related information in packets. The encapsulation of a DetNet flow allows the packets to be sent over an unique queuing technology. The DetNet forwarding nodes along the path can follow the queue scheduling carried in the packet to achieve the end-to-end bounded latency. The DetNet forwarding sub-layer may provide capabilities applying existing queuing mechanisms or traffic treatment. For example, the traffic treatment has been proposed in [draft-du-detnet-layer3-low-latency] to decrease the micro-bursts in layer3 network for low-latency traffic. The time-scheduling queuing mechanisms includes the Time Aware Shaping [IIEEE802.1Qbv] and priority-scheduling includes the Credit-Based Shaper[IEEE802.1Q-2014] with Asynchronous Traffic Shaping[IEEE802.1Qcr]. The cyclic-scheduling queuing mechanism has been proposed in [IEEE802.1Qch] and improved in . The deadline-scheduling queuing mechanism has been proposed in and improved in . The per-flow queuing mechanism includes Guaranteed-Service Integrated service (IntServ) .

1. deterministic latency information DetNet forwarding sub-layer may provide the function and technology such as multiple queuing and traffic treatment for DetNet application flows to guarantee the deterministic latency. The DetNet data plane may also encode the deterministic latency related information in packets. The information ensuring deterministic latency should be provided for enhanced data plane as defined in and . For example, the encapsulation of a DetNet flow allows the packets to be sent over an unique queuing mechanism. It is required to carry queuing related information in data plane so as to make appropriate packet forwarding and scheduling decisions to meet the time bounds.

An IP data plane may operate natively or through the use of an encapsulation. IP encapsulation can satisfy enhanced DetNet requirements. Explicit inclusion of the flow identification, path selection, queuing and traffic treatment is possible through the use of IP options, IP extension headers or existing IP headers. For example, the queuing information has been carried in IPv6/SRv6 networks as defined in . MPLS provides a service sub-layer for traffic by adding specific flow attributes (S-label and d-cw) in packets. MPLS provides a forwarding sub-layer for traffic over implicit and explicit paths such as F-Labels. Explicit inclusion of queuing and traffic treatment is possible through the use of MPLS metadata or MPLS TC field as defined in and .

As described in section 3.6.1, it is required to support the configuration of multiple queuing mechanisms. Different queuing mechanisms may be supported at different levels of latency, jitter and other guarantees. The type of queuing mechanism and the related queuing parameters should be advertised and configured. For example, the deterministic links with queuing resource could be distributed by IGP protocol as per . And the queuing parameters are carried in deterministic latency information may be selected in path computation as per .

The deterministic routes may be loose routes in distributed scenarios. It is required to support the distributed deterministic routes which are established by distributed protocols such as IGP as defined in .

In large-scale deterministic networks, it may across multiple network domains, it is required to support the inter-domain deterministic routes to achieve the end-to-end latency, bounded jitter. And the deadline of latency and jitter of each domain and segment should be determined and controlled. The inter-domain mechanism MUST be considered at the boundary nodes such as BGP configurations defined in .

As defined in , the deterministic latency constraints can be carried in PCEP extensions and the end-to-end deterministic path computation should be achieved for DetNet service.

As defined in , the BGP flowspec can be used for the filtering of the packets that match the DetNet networks and the mapping between TSN streams and DetNet flows in the control plane.

TBA

The authors would like to thank Peng Liu, Bin Tan, Aihua Liu Shaofu Peng for their review, suggestions and comments to this document.

TBA