]> Amazon Web Services

440 Terry Ave N Seattle WA 98109 USA nsd.ietf@gmail.com

TSV TCPM Working Group Internet-Draft Reno-based congestion control has been referred as the standard document from IETF for long time that describes congestion control principle of the Internet. In the meantime, IETF recently has published two new congestion control standards that use slightly different schemes from the previous one. This document provides analysis for the differences between these standards in order to provide helpful information when an unified congestion control principles for the Internet is standardized in the future.

Introduction specifies Reno-based congestion control and it has been referred as the standard document from IETF that outlines congestion control principle of the Internet. On the other hand, IETF recently have published two new congestion control standards; as Reno-based congestion control for QUIC and as CUBIC congestion control for various transport protocols. We believe all transport protocols should share the same congestion control principle so that they can share network resources mostly fairly. From this point, we believe the concepts described in these standards should not conflict each other. In our study, the new standards mostly follow the principles described in , however, there are certain differences in their schemes or the constant values, which may create certain performance differences. This document provides a list of such differences as a result of our study, but does not provide any evaluations nor analysis for the performance impacts by them. Hence, some differences described in the document might be proved to be negligible in further analysis. Or, others may be considered to create distinct performance differences so that they will need to be updated to avoid conflicts between the standards. However, given that the scale of the Internet, we think such evaluations will not be easy as it would require large-scale and long-term analysis. The aim of the document is to simply describe the differences in them and discuss their potential impacts as a reference for further analysis. We hope the document will be an useful resource when an unified congestion control principles for the Internet is needed to be standardized in the future.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Analysis for differences between RFC9002 and RFC5681 We think there are the following differences between RFC9002 and RFC5681.

Initial Window RFC5681 specifies Initial Window to be at most 4 segments or 4380 bytes as specified in while RFC9002 allows it to be up to 10 segments or 14720 bytes. allows TCP connections to use up to 10 segments or 14600 bytes for Initial Window, however RFC5681 does not adopt it as it is an experimental document. The difference in the choice of Initial Window will have certain impacts on the growth rate of congestion window.

Loss Window RFC5681 specifies Loss Window Size to be 1 segment while RFC9002 uses 2 segments for it. As Section 4.8 in RFC9002 describes, using 2 segments for Loss Window can reduce the chance of RTO and avoid additional delays caused by delayed ack algorithm. However, this could mean when an RFC5681 connection and an RFC9002 connection get RTOs at the same time, the RFC9002 connection can recover congestion window size more than 2 times faster than the RFC5681 connection.

SlowStart Threshold After Packet loss In RFC9002, a sender sets the slow-start threshold to the half value of the congestion window when packet loss is detected. However, in RFC5681, it uses the half value of the flight size instead of congestion window. As there can be some situations where there is a significant difference between flightsize and congestion window, the choice of them will have considerable impacts. Even more, RFC5681 implicitly disallows to use congestion window here as it states:

• " Implementation Note: An easy mistake to make is to simply use cwnd, rather than FlightSize, which in some implementations may incidentally increase well beyond rwnd. "

We fathom this sentence stemmed from the fact that there were some implementations which increment congestion window on every acknowledgment even though receiver's window was fully subscribed in the past. However, In RFC9002, it is prohibited to increase congestion window when it is underutilized to prevent this situation (Section 7.8). RFC9002 also allows to use other mechanisms to update its congestion window during idle periods such as .

Window Growth in Slow Start Period In RFC9002, when a sender is in slow start, the congestion window increases by the number of bytes acknowledged on each acknowledgment segment arrival. On the other hand, RFC5681 increases congestion window by at most 1 full segment. RFC5681 mentions RFC3465 which uses similar method to RFC9002, however RFC5681 does not recommend to use it. In addition, RFC3465 defines the limited factor: L which controls the aggressiveness of the algorithm. RFC3465 recommends to use L=2. This means it can allow to increase congestion window by at most 2 full segments. This logic in RFC5681 is overridden by when hystart++ is activated in the connection. When hystart++ is enabled, as long as some packet pacing mechanisms are used in the connection, congestion window can be increased by the amount of acknowledged data in an ACK packet like RFC9002. However, the increase of congestion window per ACK is limited to 8 SMSS if there is no pacing. The performance differences in this logic will be more visible in the presence of stretch ACKs.

Loss Recovery Algorithm The gist of the loss recovery algorithm in RFC5681 is to retransmit all lost segments found in the previous round trip and once all of them have been acknowledged, it migrates to Congestion Avoidance from Recovery period. The detailed algorithms are specified in and . On the other hand, RFC9002 specifies the ends of Recovery period as when one of any packets sent during the Recovery period is acknowledged. This means RFC9002 can end Recovery period even not all lost segments in the previous round trip has been successfully retransmitted. Moreover, it can end Recovery period even if some segments have been lost during Recovery period as long as one or more packets have been acknowledged. Although we think this behavior will not lead to a congestion collapse, it looks more aggressive than RFC5681. For example, when there is a congestion where some but not all segments have been lost during several round trips, RFC5681 reduces congestion window by half every round trip (as long as retransmission schemes work successfully, otherwise it will be timed out). On the other hand, RFC9002 will repeat Recovery period and Congestion Avoidance Period in turn, which reduces congestion window by half every other round trip. Another aspect of the loss recovery in RFC9002 is persistent congestion that is equivalent to TCP's RTO. In RFC9002, data sender establishes persistent congestion only when all sent packets are lost for a long enough duration. This period is equivalent to the duration for an RTO and two TLPs in TCP. This will mean RFC9002 reduces congestion window to minimal value only when there is a extreme severe congestion. On the other hand, RFC5681 has more chances for RTOs as it gets RTOs when fast retransmission/fast recovery scheme doesn't work due to insufficient number of acknowledgments.

Loss Detection Schemes In RFC9002, in addition to acknowledgment-based loss detection scheme which is also defined in RFC5681, it specifies another loss detection scheme similar to RACK-TLP. Although RACK-TCP is a standard document, RFC5681 has no description for it.

Minimal RTO Values RFC9002 follows most of the parts of which defines the standard algorithm of retransmission timer computations and managements for TCP. However, it does not follow 1 sec for minimal RTO values in RFC6298 as it does not specify minimal RTO. This might not be a major problem because it is known that various TCP implementations already adopt lower values for minimal RTO. In addition, QUIC has more explicit mechanism to identify spurious RTOs than TCP, hence we believe there is no risk for large-scale network issues in this. However, the impacts for not having minimum RTO is still an important research topic for the performance and efficiency of transport protocols.

Analysis for differences between CUBIC and RFC5681 We think the following points in the CUBIC specification can cause differences behavior from RFC5681.

Multiplicative Window Decrease Factor The CUBIC specification uses 0.7 for Multiplicative Window Decrease factor while RFC5681 uses 0.5. We think the rationale for using 0.5 in RFC5681 is derived from the following sentences in , hence we presume using 0.7 instead will not be too aggressive to lead to a congestion collapse. However, it can still be more aggressive than RFC5681 which may cause unfair resource sharing.

• " We usually run our nets with ≤ 0.5 so it’s probable that there are now exactly two conversations sharing the bandwidth. I.e., you should reduce your window by half because the bandwidth available to you has been reduced by half. And, if there are more than two conversations sharing the bandwidth, halving your window is conservative "

In order to compensate the aggressiveness by using the aggressive decrease factor, CUBIC uses "Reno-friendly model" which employs slower window growth rate in low BDP environments. We will discuss the validity of the model in the following section, however, even if the model is valid, CUBIC can be more aggressive than RFC5681 in some situations. For example, when there are a congestion that can last several round trips, CUBIC reduces congestion window by 30% every round trip while RFC5681 reduces it by half. In this situation, the window decrease rate for CUBIC will be mostly the half of RFC5681's. In addition, during Recovery period, CUBIC transmits data with 70% of the previous congestion window size while RFC5681 uses 50% of it. Another example is that congestion window size can be much larger than network capacity during slow start and CUBIC's high decrease factor may have more impacts than RFC5681. For example, let's say network capacity is 100Mbps and a TCP sender's congestion window size at a certain round trip allows to transfer data at 99Mbps, which is lower than the capacity. If this sender is in slow-start, the congestion window size may glow to transfer data at 198Mbps in the next round trip and can cause many packet losses. In case of RFC5681, congestion window will be reduced by half in the following round trip and transfer rate will be 99Mbps. However, in case of CUBIC, the transfer rate will be 138.6Mbps which still exceeds the network capacity. This would mean CUBIC can saturate network for two round trips in this example while RFC5681 does only for one round trip. However, this might be a pathological example since many recent transport stacks support pacing mechanism and or to mitigate the overshooting during slow-start.

Reno-friendly Model CUBIC employs Reno-friendly model which is designed to be fair to RFC5681 in low BDP environments. The model in CUBIC is derived from and it is based on AIMD congestion control as same as RFC5681, but adopts different increase factor and multiplicative factor . In RFC5681, is 1.0 and is 0.5. This means RFC5681 increases congestion window by 1 segment per acknowledgment when a transport protocol is in congestion avoidance and reduces congestion window by half when packet losses are detected. In the meantime, in {I-D.ietf-tcpm-rfc8312bis}}, is around 0.529 and is 0.7. Hence, CUBIC reduces congestion window less than RFC5681 at packet loss, but at the same time, it reduces the window growth rate so that the performance of CUBIC and RFC5681 will be mostly the same. We explain the rationale behind the values for and for CUBIC in , but the important point for using these values is that it is based on the following two presumptions.

• CUBIC connection(s) and Reno connection(s) are sharing the same network resource
• When packet losses happen, both CUBIC connection and Reno connection lost packets at the same time.

Although the first presumption can be considered as a common situation, the second presumption will not be a common one as there should be various patterns in packet losses. Moreover, in this model, CUBIC increases congestion window by 0.529 segments per acknowledgment which means CUBIC transmits a segment upon the arrival of every two acknowledgments because a transport protocol usually does not send a segment until it has at least one full segment space available in congestion window. This makes harder to establish the second presumption even more. This might be a relatively minor point that do not have significant impacts on overall performance of the model, however, more detailed analysis with realistic packet dynamics will be desirable. A recent report on this point shows that the model in CUBIC looks mostly fair to RFC5681 in low BDP environments .

SlowStart Threshold After Packet loss In CUBIC, a sender sets the slow-start threshold to the half value of the FlightSize just like specified in RFC5681. However, it is not prohibited to use congestion widow instead. In addition, CUBIC mentions employing as a more sophisticated approach.

Security Considerations TODO Security

IANA Considerations This document has no IANA actions.

References Normative References This document describes loss detection and congestion control mechanisms for QUIC. This document defines TCP's four intertwined congestion control algorithms: slow start, congestion avoidance, fast retransmit, and fast recovery. In addition, the document specifies how TCP should begin transmission after a relatively long idle period, as well as discussing various acknowledgment generation methods. This document obsoletes RFC 2581. [STANDARDS-TRACK] University of Nebraska-Lincoln University of Colorado at Boulder Bowery Farming Apple Inc. NetApp CUBIC is a standard TCP congestion control algorithm that uses a cubic function instead of a linear congestion window increase function to improve scalability and stability over fast and long- distance networks. CUBIC has been adopted as the default TCP congestion control algorithm by the Linux, Windows, and Apple stacks. This document updates the specification of CUBIC to include algorithmic improvements based on these implementations and recent academic work. Based on the extensive deployment experience with CUBIC, it also moves the specification to the Standards Track, obsoleting RFC 8312. This also requires updating RFC 5681, to allow for CUBIC's occasionally more aggressive sending behavior. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References Confluent Microsoft Microsoft This document describes HyStart++, a simple modification to the slow start phase of congestion control algorithms. Slow start can overshoot the ideal send rate in many cases, causing high packet loss and poor performance. HyStart++ uses increase in round-trip delay as a heuristic to find an exit point before possible overshoot. It also adds a mitigation to prevent jitter from causing premature slow start exit. This document proposes an experiment to increase the permitted TCP initial window (IW) from between 2 and 4 segments, as specified in RFC 3390, to 10 segments with a fallback to the existing recommendation when performance issues are detected. It discusses the motivation behind the increase, the advantages and disadvantages of the higher initial window, and presents results from several large-scale experiments showing that the higher initial window improves the overall performance of many web services without resulting in a congestion collapse. The document closes with a discussion of usage and deployment for further experimental purposes recommended by the IETF TCP Maintenance and Minor Extensions (TCPM) working group. This document proposes a small modification to the way TCP increases its congestion window. Rather than the traditional method of increasing the congestion window by a constant amount for each arriving acknowledgment, the document suggests basing the increase on the number of previously unacknowledged bytes each ACK covers. This change improves the performance of TCP, as well as closes a security hole TCP receivers can use to induce the sender into increasing the sending rate too rapidly. This memo defines an Experimental Protocol for the Internet community. This document defines the standard algorithm that Transmission Control Protocol (TCP) senders are required to use to compute and manage their retransmission timer. It expands on the discussion in Section 4.2.3.1 of RFC 1122 and upgrades the requirement of supporting the algorithm from a SHOULD to a MUST. This document obsoletes RFC 2988. [STANDARDS-TRACK] RFC 5681 documents the following four intertwined TCP congestion control algorithms: slow start, congestion avoidance, fast retransmit, and fast recovery. RFC 5681 explicitly allows certain modifications of these algorithms, including modifications that use the TCP Selective Acknowledgment (SACK) option (RFC 2883), and modifications that respond to "partial acknowledgments" (ACKs that cover new data, but not all the data outstanding when loss was detected) in the absence of SACK. This document describes a specific algorithm for responding to partial acknowledgments, referred to as "NewReno". This response to partial acknowledgments was first proposed by Janey Hoe. This document obsoletes RFC 3782. [STANDARDS-TRACK] This document presents a conservative loss recovery algorithm for TCP that is based on the use of the selective acknowledgment (SACK) TCP option. The algorithm presented in this document conforms to the spirit of the current congestion control specification (RFC 5681), but allows TCP senders to recover more effectively when multiple segments are lost from a single flight of data. This document obsoletes RFC 3517 and describes changes from it. [STANDARDS-TRACK] This document presents the RACK-TLP loss detection algorithm for TCP. RACK-TLP uses per-segment transmit timestamps and selective acknowledgments (SACKs) and has two parts. Recent Acknowledgment (RACK) starts fast recovery quickly using time-based inferences derived from acknowledgment (ACK) feedback, and Tail Loss Probe (TLP) leverages RACK and sends a probe packet to trigger ACK feedback to avoid retransmission timeout (RTO) events. Compared to the widely used duplicate acknowledgment (DupAck) threshold approach, RACK-TLP detects losses more efficiently when there are application-limited flights of data, lost retransmissions, or data packet reordering events. It is intended to be an alternative to the DupAck threshold approach. This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. This document describes HyStart++, a simple modification to the slow start phase of congestion control algorithms. Slow start can overshoot the ideal send rate in many cases, causing high packet loss and poor performance. HyStart++ uses increase in round-trip delay as a heuristic to find an exit point before possible overshoot. It also adds a mitigation to prevent jitter from causing premature slow start exit. This document provides a mechanism to address issues that arise when TCP is used for traffic that exhibits periods where the sending rate is limited by the application rather than the congestion window. It provides an experimental update to TCP that allows a TCP sender to restart quickly following a rate-limited interval. This method is expected to benefit applications that send rate-limited traffic using TCP while also providing an appropriate response if congestion is experienced. This document also evaluates the Experimental specification of TCP Congestion Window Validation (CWV) defined in RFC 2861 and concludes that RFC 2861 sought to address important issues but failed to deliver a widely used solution. This document therefore reclassifies the status of RFC 2861 from Experimental to Historic. This document obsoletes RFC 2861.

Acknowledgments add people in tcpm-wg community.

Appendix A: Deriving increase factor for CUBIC from AIMD model This section describes how increase factor: used in CUBIC is determined from AIMD congestion control model. We define AIMD(, ) as AIMD congestion control that uses an increase parameter an a decrease parameter . Hence, AIMD(1, 0.5) represents congestion control described in RFC5681 while CUBIC can be expressed as AIMD(, ) where =0.7. We also define Wmax as the congestion window size that can fully utilize network capacity. At first, it is clear that =0.7 is more aggressive than =0.5 when there is no other traffic. As cwnd growth is linear, if there's enough long time for the data transfer, the average congestion window for =0.5 will be (1.0 + 0.5)/2 = 0.75 Wmax and it will be (1.0 + 0.7)/2 = 0.85 Wmax for =0.7. Hence, it is obvious that this model does not aim for the cases where there's no other traffic. The choice of (, ) for CUBIC is designed to be fair only when it competes with AIMD(1.0, 0.5). Here, we define W1 as the max window size for AIMD(1.0, 0.5) and W2 as the max window size for CUBIC's AIMD(=X, =0.7) when they compete each other. In this situation, AIMD(1.0, 0.5) model oscillates between 0.5 W1 and 1.0 W1 in a congestion epoch. CUBIC's AIMD (=X, =0.7) model oscillates between 0.7 W2 and 1.0 W2 in the same congestion epoch. When these two models have the same loss ratio, it should satisfy the following equation (1). Also, in one congestion epoch, (=1.0, =0.5) increases congestion window by 0.5 W1 while (=X, =0.7) increases 0.3 W2. The length of congestion epoch for AIMD(1.0, 0.5) can be expressed as 0.5 W1/1.0 and it will be expressed as 0.3 W2 /X for AIMD(=X, =0.7). Because AIMD(1.0), AIMD(=X, =0.7) should have the same congestion epoch when they compete equaly, it should satisfy the following equation (2). From equations (1) and (2), we get X=0.529.

Contributors The contents in this documents are the individual contributions from the authors and do not relate to the authors' positions at their affiliations.