Deutsche Telekom

Deutsche-Telekom-Allee 9 Darmstadt 64295 Germany Markus.Amend@telekom.de

Deutsche Telekom

Deutsche-Telekom-Allee 9 Darmstadt 64295 Germany dirk.von-Hugo@telekom.de

IRTF ICCRG Working Group Internet-Draft This document discusses the issue of packet reordering which occurs as a specific problem in multi-path connections without reliable transport protocols such as TCP. The topic is relevant for devices connected via multiple accesses technologies towards the network as is foreseen, e.g., within Access Traffic Selection, Switching, and Splitting (ATSSS) service of 3rd Generation Partnership Project (3GPP) enabling fixed mobile converged (FMC) scenario.

Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at . Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 28 April 2022.

Copyright Notice Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents () in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• .  Introduction
• .  State of the Art
• .  Problem Statement
• .  Scheduling mechanisms
• .  Resequencing mechanisms
  • .  Passive
  • .  Exact
  • .  Static Expiration
  • .  Adaptive Expiration
  • .  Delay Equalization
  • .  Fast Packet Loss Detection
    • .  Connection sequencing
    • .  Per-path sequencing
    • .  Combination connection and per-path sequencing
• .  Recovery mechanisms
  • .  FEC (Forward Error Correction)
  • .  Network Coding
• .  Retransmission mechanisms
  • .  Signaling
  • .  Anticipated
  • .  Flow-selection
  • .  Other re-transmission issues
• .  Security Considerations
• .  Informative References
• Authors' Addresses

Introduction Mobile end user devices nowadays are mostly equipped with multiple network interfaces allowing to connect to more than one network at a time and thus increase data throughput, reliability, coverage, and so on. Ideally the user data stream originating from the application at the device is split between the available (here: N) paths at the sender side and re-assembled at an intermediate aggregation node before transmitted to the corresponding host in the network as depicted in .

Reference Architecture for multi-path reordering

However, when several paths are utilized concurrently to transmit user data between the sender and the receiver, different characteristics of the paths in terms of bandwidth, delay, or error proneness can impact the overall performance due to delayed packet arrival and need for re-transmit in case of lost packets. Without further arrangements the original order of packets at the sending UE side is no longer maintained at the receiving host and a reordering or re-arrangement has to occur before delivery to the application at the far end site. This can be performed at earliest at the aggregation node with a minimum additional delay due to re-transmission requests or at latest either by the application on the host itself or the transmission protocol. It is a goal of the present document to collect and describe mechanisms to maintain the sequence of split traffic over multiple paths. These mechanisms are generic and not dedicated to a specific multipath network protocol, but give clear guidance on requirements and benefits to maintainers of multipath network protocols.

State of the Art Regular TCP protocol offers such mechanism with queues for in-order and out-of order (including damaged, lost, duplicated) arrival of packets. This is also provided by MPTCP as the first and successful Multipath protocol which however also requires new methods as sequence numbers both on (whole) data (stream) and subflow level to ensure in-order delivery to the application layer on the receiver side . Moreover, careful design of buffer sizes and interpretation of sequence numbers to distinguish between (delayed) out-of-order packets and completely lost ones has to be considered. already reflects on proper packet scheduling schemes (at the sender side) to reduce the effort for re-assembly or even make such (time consuming) treatment unnecessary. MP-QUIC introduces the concept of uniflows with own IDs claiming to get rid of additional sequence numbers for reordering as required in Multipath TCP . Although admits that statistical performance information should help a host in deciding on optimum packet scheduling and flow control a dedicated packet scheduling policy is out of scope of that document. A further improvement versus MPTCP can be achieved by decoupling paths used for data transmission from those for sending acknowledgments (ACKs) or claiming for re-transmission by NACKs to not introduce further latency. specifies algorithms for QUIC Loss Detection and Congestion Control by using measurement of Round Trip Time (RTT) to determine when packets should be retransmitted. Draft proposes to enable one way delay (1WD) measurements in QUIC by defining a TIME_STAMP frame to carry the time at which a packet is sent and combine the ACKs sent with a timestamp field and thus allow for more precise estimation of the (one-way) delay of each uniflow, assisting proper scheduling decisions. Also other protocols as Multi-Access Management Services (MAMS) consider the need for reordering on User Plane level which may be done at network and client level by introducing a new Multi-Access (MX) Convergence Layer. introduces accordingly Traffic Splitting Update (TSU) messages and Packet Loss Report (PLR) messages including beside others Traffic Splitting Parameters and an expected next (in-order) sequence number, respectively. on Generic Multi-Access (GMA) Convergence Encapsulation Protocols introduces a trailer-based encapsulation which carries one or multiple IP packets or fragments thereof in a Protocol Data Unit (PDU). At receiver side PDUs with identical Sequence Numbers (in the trailer) are to be placed in the relative order indicated by a so-called Fragment Offset.

Problem Statement Assuming for simplicity the minimum multipath scenario with two separate paths for transmission of a flow of packets with sequence numbers (SN) SN1 ... SM. In case the scheduling of packets is done equally to both paths and path 2 exhibits a delay of the duration of transmission time required for, e.g., two packets (assuming fixed packet size and same constant data for both paths) for an exemplary App-originated sequence of packets as SN1 SN2 SN3 SN4 SN5 SN6 SN7 SN8 ... the resulting sequence of packets could look as depicted in which of course depends on the queue processing and buffering at the Aggregation Proxy.

Exemplary data transmission for a dual-path scenario |path 1 SN1 SN3 SN5 SN7...| | | |------------------------>| | | |path 2 SN2 SN4 SN6 SN8...| | | |------------------------>| | | | |SN1 SN3 SN2 SN5 SN4 SN7| | | |======================>| | | | | ]]>

In such a case reordering at the Aggregation Node would be simple and straight forward. It even could be avoided if the scheduling would already take the expected different delays into account (e.g. by pre-delaying the traffic on path 1 thus of course not leveraging the lower delay). Different from this simplistic scenario in general the data rate on both paths will vary in time and be not equal, also different and variable latency (jitter) per path will be introduced and in addition loss of packets as well as potential duplication may occur making the situation much more complicated. In case of loss detection after a threshold waiting time a retransmission could be initiated by the Host or if possible already by the Aggregation Node. Alternatively the UE could send redundant packets in advance coded in such a way that it allows for derivation of, e.g., one lost packet per M correctly received ones or by a (real-time) application able to survive singular lost packets. Holding multiple queues and a large enough buffer both at UE and at the Aggregation Node would be required to apply proper scheduling at UE and reordering during re-assembly at Aggregation Node to mitigate the sketched impact of multiple paths' variable characteristics in terms of transmission performance. ...

Scheduling mechanisms Scheduling mechanisms decide on sender side how traffic is distributed over the paths of a multipath-setup. gives an overview of possible distribution schemes. For this document it is assumed, that schedulers are used, which simultaneously distribute traffic over more than one path, whereas path characteristics differ between those multiple paths (e.g. a latency difference exists). While on the one hand, the traffic scheduling causes out-of-order multipath delivery when simultaneously utilize heterogeneous paths, it can also be used to mitigate this problem. Pre-delaying data on a fast path, according to the latency difference of the slowest path is aimed, e.g., by OTIAS , DAPS , and BLEST . However, the success is much dependent on the accuracy of path information like path latency, throughput, and packet loss rate. In heterogeneous and volatile environments most often such information have to be estimated, e.g., using congestion control. That means, it takes at least one RTT to gain first indications and probably several RTTs to converge to a worthwhile accuracy. Changes of path characteristics in sub-RTT time frames put such a system to test. Dependent on the demand on in-order delivery and/or the accuracy of the relevant path information, scheduling might be an exclusive alternative or can be applied in conjunction with other discussed mechanisms in this document. proposes to use a predictive Adaptive Order Prediction Scheduling (AOPS) mechanism considering both the anticipated time of packet delivery and the reliability of each path to optimize the traffic scheduling for MP-DCCP, thus coping with reordering and achieving in-order delivery. Scheduling will not help to overcome any degree of out-of-order delivery, when the scheduling goal is different to this. For example a strict cost prioritization of Wi-Fi over cellular access in a mobile phone might be assumed counterproductive.

Resequencing mechanisms Resequencing mechanisms are responsible to modify the sequence of received data split over multiple paths according to a sequencing scheme. The degree of resequencing can reach from no measure up to re-generating the exact order. Typically at least one sequencing scheme, describing the order of how data was generated on sender side is prerequisite. This is referred to as "connection sequencing". Under certain circumstances an additional sequencing scheme per path of the multi-path setup can be leveraged, to optimize packet loss detection and is further elaborated in . For most multipath protocols both sequencing schemes are already available. Packet loss detection becomes important when multipath protocols are applied which do not guarantee successful transmission as TCP achieves by acknowledgement of successful reception. For example, or the combination of and are unreliable protocols in that sense. For simplicity all the mechanism described in the following are explained based on two paths but in principle would work with any other amount though.

Passive This approach includes no active change or reordering at the transport level and purely re-combines the packet flows incoming from both paths as is. All modification of the resulting sequence of packets is left to the application at the end node. Here no processing delay is added due to the resequencing but since no early packet loss detection with subsequent re-transmission request on transport level is possible the risk of a larger delay due to late loss detection at the application will arise in case of lossy connections.

Exact This approach covers all mechanisms which attempt to re-generate the original order of packets in the flow exactly, independent of the expected or resulting delay due to waiting time for all packets on all paths to arrive. In case of unreliable transport protocols this may result in a large delay due to Head-of-Line blocking and for actual packet loss in a remaining packet gap which causes a stand still without an option to recover. For applications demanding near real-time delivery of packets it should not be applied.

Static Expiration This method to detect and decide on packet loss assumes a certain fixed time threshold for the gap between packets within a sequence after re-combination of both paths. A possible re-transmission - either in the multipath system internally or based on the piggybacked protocol/service - will possibly not be requested before this threshold is exceeded. Thus an additional delay in the overall latency budget will occur so that this simple approach is only recommended for non-time critical applications. Every packet loss or simultaneous transmission of data over the short and long latency path will cause spikes in the service perceived latency.

Adaptive Expiration Here the packet gap is assumed as packet loss after exceeding a flexibly decided on time threshold which may be derived dynamically from the differences between latencies both paths exhibit. As the latency may vary due to propagation conditions or routing paths this latency difference has to be monitored and statistically evaluated (smoothed) which introduces additional effort. A possible solution for this is the determination of the the one way latency as described in or sending available RTT information from the sender from which the receiver can calculate the latency difference.

Delay Equalization This is an ordering mechanism which delays data forwarding on the faster path by the latency difference to the slower path. Ideally the resequencing effort on the aggregated packet flow can be greatly reduced up to no resequencing at all. Due to time variation in path delays (jitter) and delay differences and the required time for decision and feedback on the delay, some re-sequencing still remains to be executed. Similar to , explicit knowledge of the latency difference is required. Strictly speaking this method allows to avoid resequencing based on sequencing information. However, the overall delay may be larger since the advantage of the short-delay path is not exploited. In combination with or resequencing can be added with a presumably lower resequencing effort to scenarios without delay equalization. The essence is a in-order stream with a unified latency across the multiple paths.

Fast Packet Loss Detection The following sections describe methods to achieve unambiguous detection of packet loss independent from thresholds in or . Furthermore, packet loss can be differentiated from delayed delivery. The benefit is a much faster decision plane based on monitoring the sequence space of consecutive packets. For that, the sequencing coming along with the receiver based re-sequencing is further leveraged. Two sequencing schemes are considered here, the connection and the per-path sequencing.

Connection sequencing Connection sequencing marks the outgoing packets in the order they enter the multipath system and is independent from a particular selected path for transmission. After arrival at the aggregation node the lowest packet sequence number at each of the multiple paths is compared the that of the last correctly received packet. When the numbers are not consecutive (i.e., when on all paths a higher number is received than the next expected in-order packet), an overall packet loss is detected. While only a single comparison of packet numbers has to be performed and the out-of-order arrival on a single path can be partly compensated this scheme does not allow for immediate detection of where reordering happens.

Per-path sequencing Per-path sequencing is a path inherent sequencing mechanism valid in the particular path domain only. In this case the packets are marked by path-specific sequence numbers at the sender side and at each interface of the aggregation node the sequence numbers of arriving packets are compared on per-path level. When a higher sequence number is received than the one which is waited for (next expected in-order packet), a packet loss for this specific path is declared. This may prevent partly misinterpretation of out-of-order arrival as packet loss and allow for path specific countermeasures towards overall performance improvement, as, e.g., chosing a more robust transmission technique for this path.

Combination connection and per-path sequencing While the benefits from the individual sequencing schemes above can be combined, a further benefit crystallizes. Since the out-of-order arrival is detected on per-path basis, the path specific out-of-order delivery rate can be used as a criterion to choose repair parameters
on a per-path basis (which thus may work more efficiently). In addition the decision on the path selection and weighting can be made based on this criterion. Thus an improved overall performance can be achieved in this case. [to be checked/continued...]

Recovery mechanisms Recovering packets, in particular lost packets or assumed lost packets on receiver side avoids re-transmission and potentially mitigates the resequencing process in respect to detecting packet loss. Shorter latencies will be an expected outcome. Discussing the complexity, computation overhead and reachable benefit is subject of this section.

FEC (Forward Error Correction) This approach is based on introduction of redundancy to user data to detect errors and subsequently reconstruct data in case of a limited number of bit or Byte errors. As such packet with corrupted data can be recovered up to a certain degree but in case of a too high bit error rate (BER) a packet is completely lost. However, in combination with scrambling, i.e. the sequence of original data stream is distributed over multiple packets and re-compiled afterwards also data from lost packets could be recovered. As such methods introduce additional delay and overhead it is mainly applied in case of long re-transmission delays as, e.g., is typical for satellite transmission. FEC can be applied to each path separately (e.g., if they exhibit deviating performance characteristics to not degrade the 'better one') or in an overall FEC fashion before split and recombination which would support scrambling and facilitate recovery of completely lost packets on the 'worse path'. Unsuccessful application of FEC may enable quick detection of unrecoverable errors in a packet and thus trigger re-transmission from the sender side before time-out.

Network Coding In linear network coding (LNC) network nodes (or interfaces of a device) do not simply relay the packets of information they receive, but combine several packets together for transmission. After reception of combined and separate packets the maximum possible information flow in a network can be detected and throughput, efficiency and scalability, as well as resilience to attacks and eavesdropping can be improved. The method in general improves with the number of paths in excess of two. According to drawbacks of LNC are high decoding computational complexity, high transmission overhead, and linear dependency among coefficients vectors for en- and decoding. Triangular network coding (TNC) addresses the high encoding and decoding computational complexity without degrading the throughput performance, with code rate comparable to that of LNC. TNC is therefore advantageous for implementation on small devices mobile phones and wireless sensors with limited processing capability and power supply .

Retransmission mechanisms Re-transmission becomes interesting when it can help to reduce the time spent on waiting for outstanding packets for re-sequencing. In particular scenarios when for example a known path RTT (Round Trip Time) lets expect a shorter time to re-transmit than wait for packet loss detection, a likely scenario in, e.g., . It could also avoid a potential late triggering of re-transmission by the end-to-end service. On the other hand for sake of resource efficiency the amount of unnecessary retransmissions should be limited to not degrade the overall throughput of the connection.

Signaling In case of detected packet loss the receiver has to send a corresponding signalling message to the sender to re-transmit a missing packet. This is the traditional way of negative acknowledgement in case of missing the correct reception of packets
within a time window and sending a repeat-request. This approach requires a send buffer which keeps information for a reasonable time, thus allowing the beneficial use of this mechanism. On the other
hand the additional delay in terms of at least once the RTT until the
lost packet is correctly received results in performance degradation
for time-critical applications. ... [to be continued?]

Anticipated To speed up the induced re-transmission delay a pro-active or anticipated approach would allow to trigger the sender to re-transmit data without needing to wait for notification from the receiver. This method can be applied when the assumed packet loss can be estimated based on other data, e.g., from lower layer, such as information on path or
link quality degradation derived from, e.g., an increased raw BER
detected by FEC mechanism (see ).
[to be continued/extended?]

Flow-selection Repeating data on the same path is not always useful. In some scenarios it makes sense to re-transmit data on another path, e.g., when the original path is broken or another path is known to provide higher throughput or lower packet loss. To apply such a selection of the flow for re-transmission.

• Requires path independent identification of data, e.g., the connection sequencing
• Has to consider MTU discrepancies between paths

Flow selection for re-transmitting data can be combined with detection mechanisms as described in or .

Other re-transmission issues In certain scenarios data to be re-transmitted can be duplicated across paths (either in advance or after loss detection) to increase reliability and reduce potential overall transmission delay.
However, such approaches decrease the resource efficiency and reduce
the overall user throughput. A more pro-active measure would
be to encode multiple packets either on per-path or on per-connection
basis in a single 'repair packet' in 'XOR style' to be injected after
a set of packets (similarly as described in . This would allow
to recreate exactly one lost packet out of the set in case the others
have been correctly received. Depending on the anticipated loss rate
the amount of packets within a set is chosen to more efficiently use
the transmission resources. [to be continued]

Security Considerations This document does not add any additional security considerations in
addition to the ones introduced by multipath extensions to other
transmission protocols as, e.g., described for MPTCP in .
Also the described issues for GMA , MP-DCCP
, and MP-QUIC
may apply here.

Informative References Deutsche Telekom Deutsche Telekom Karlstad University Karlstad University City University of London BT DCCP communication is currently restricted to a single path per connection, yet multiple paths often exist between peers. The simultaneous use of these multiple paths for a DCCP session could improve resource usage within the network and, thus, improve user experience through higher throughput and improved resilience to network failures. Use cases for a Multipath DCCP (MP-DCCP) are mobile devices (handsets, vehicles) and residential home gateways simultaneously connected to distinct paths as, e.g., a cellular link and a WiFi link or to a mobile radio station and a fixed access network. Compared to existing multipath protocols such as MPTCP, MP- DCCP provides specific support for non-TCP user traffic as UDP or plain IP. More details on potential use cases are provided in [website], [slide] and [paper]. All this use cases profit from an Open Source Linux reference implementation provided under [website]. This document presents a set of extensions to traditional DCCP to support multipath operation. Multipath DCCP provides the ability to simultaneously use multiple paths between peers. The protocol offers the same type of service to applications as DCCP and it provides the components necessary to establish and use multiple DCCP flows across potentially disjoint paths. Work in Progress UCLouvain UCLouvain UCLouvain Tessares Apple Deutsche Telekom This document proposes a series of abstract packet schedulers for multipath transport protocols equipped with a congestion controller. Work in Progress UCLouvain UCLouvain This document specifies extensions to the QUIC protocol to enable the simultaneous usage of multiple paths for a single connection. These extensions are compliant with the single-path QUIC design and preserve QUIC privacy features. Discussion about this draft is encouraged either on the QUIC IETF mailing list quic@ietf.org or on the GitHub repository which contains the draft: https://github.com/qdeconinck/draft-deconinck-multipath- quic. Work in Progress Private Octopus Inc. The TIMESTAMP frame can be added to Quic packets when one way delay measurements are useful. The timestamp is set to the number of microseconds from the beginning of the node's epoch to the time at which the packet is sent. The draft defines the "enable_timestamp" transport parameter for negotiating the use of this extension frame, and the TIMESTAMP frame. Work in Progress Apple Inc. Apple Inc. Google LLC This document defines an extension to the QUIC transport protocol to add support for sending and receiving unreliable datagrams over a QUIC connection. Discussion of this work is encouraged to happen on the QUIC IETF mailing list quic@ietf.org (mailto:quic@ietf.org) or on the GitHub repository which contains the draft: https://github.com/quicwg/ datagram (https://github.com/quicwg/datagram). Work in Progress Fastly Google This document describes loss detection and congestion control mechanisms for QUIC. Work in Progress Alibaba Inc. Alibaba Inc. Private Octopus Inc. Alibaba Inc. ICT-CAS This document specifies multipath extension for the QUIC protocol to enable the simultaneous usage of multiple paths for a single connection. The extension is compliant with the single-path QUIC design. The design principle is to support multipath by adding limited extension to [QUIC-TRANSPORT]. Work in Progress Beijing Jiaotong University Beijing Jiaotong University Huawei Technologies Huawei Technologies This document discusses the use of One Way Latency (OWL) for enhancing multipath TCP (MPTCP). Several usages of OWL, such as retransmission policy and crucial data scheduling are analyzed. Two kinds of OWL measurement approaches are also provided and compared. More explorations related with OWL will be contribute to the performance of MPTCP. Work in Progress Intel Nokia A device can be simultaneously connected to multiple networks, e.g., Wi-Fi, LTE, 5G, and DSL. It is desirable to seamlessly combine the connectivity over these networks below the transport layer (L4) to improve quality of experience for applications that do not have built in multi-path capabilities. Such optimization requires additional control information, e.g., a sequence number, in each packet. This document presents a new light weight and flexible encapsulation protocol for this need. The solution has been developed by the authors based on their experiences in multiple standards bodies including the IETF and 3GPP, is not an Internet Standard and does not represent the consensus opinion of the IETF. This document will enable other developers to build interoperable implementations in order to experiment with the protocol. Work in Progress Intel Korea Telecom Nokia Huawei Today, a device can be simultaneously connected to multiple communication networks based on different technology implementations and network architectures like WiFi, LTE, and DSL. In such multi- connectivity scenario, it is desirable to combine multiple access networks or select the best one to improve quality of experience for a user and improve overall network utilization and efficiency. This document presents the u-plane protocols for a multi access management services (MAMS) framework that can be used to flexibly select the combination of uplink and downlink access and core network paths having the optimal performance, and user plane treatment for improving network utilization and efficiency and enhanced quality of experience for user applications. Work in Progress TCP/IP communication is currently restricted to a single path per connection, yet multiple paths often exist between peers. The simultaneous use of these multiple paths for a TCP/IP session would improve resource usage within the network and thus improve user experience through higher throughput and improved resilience to network failure. Multipath TCP provides the ability to simultaneously use multiple paths between peers. This document presents a set of extensions to traditional TCP to support multipath operation. The protocol offers the same type of service to applications as TCP (i.e., a reliable bytestream), and it provides the components necessary to establish and use multiple TCP flows across potentially disjoint paths. This document specifies v1 of Multipath TCP, obsoleting v0 as specified in RFC 6824, through clarifications and modifications primarily driven by deployment experience. In multiconnectivity scenarios, the clients can simultaneously connect to multiple networks based on different access technologies and network architectures like Wi-Fi, LTE, and DSL. Both the quality of experience of the users and the overall network utilization and efficiency may be improved through the smart selection and combination of access and core network paths that can dynamically adapt to changing network conditions. This document presents a unified problem statement and introduces a solution for managing multiconnectivity. The solution has been developed by the authors based on their experiences in multiple standards bodies, including the IETF and the 3GPP. However, this document is not an Internet Standards Track specification, and it does not represent the consensus opinion of the IETF. This document describes requirements, solution principles, and the architecture of the Multi-Access Management Services (MAMS) framework. The MAMS framework aims to provide best performance while being easy to implement in a wide variety of multiconnectivity deployments. It specifies the protocol for (1) flexibly selecting the best combination of access and core network paths for the uplink and downlink, and (2) determining the user-plane treatment (e.g., tunneling, encryption) and traffic distribution over the selected links, to ensure network efficiency and the best possible application performance.

Authors' Addresses Deutsche Telekom

Deutsche-Telekom-Allee 9 Darmstadt 64295 Germany Markus.Amend@telekom.de

Deutsche Telekom

Deutsche-Telekom-Allee 9 Darmstadt 64295 Germany dirk.von-Hugo@telekom.de