]> University of New Hampshire InterOperability Laboratory

US +1 603 862 3941 kouellette@iol.unh.edu

Operations and Management IPv6 Operations EH HBH DO It has become accepted that packets containing IPv6 Extension Headers, especially Hop-by-hop Options, are frequently dropped on the Internet. However, the question still remains as to why they get dropped and what type of devices may be dropping them. This document describes research conducted to isolate routers in a simple topology, with minimal configuration, and shows that router implementations alone are likely not the cause of packets containing IPv6 Extension Headers being dropped on the Internet.

Introduction It has become well known that packets containing IPv6 Extension Headers (EHs), especially Hop-by-Hop Options, are often dropped when traversing the Internet. However, the question still remains as to what are the primary contributors to these packet drops and under what circumstances are the packets dropped. Some of the potential culprits include ACLs, firewalls, routers, load balancers, and more. It also is not known whether these packets are dropped intentionally (e.g., ACLs) or unintentionally (e.g., firmware bugs). This document describes the research conducted and results of isolating routers to better understand what role they may play, if any, in the dropping of EHs. Specifically, this research is aiming to understand whether or not router implementations may be the cause of dropped packets containing EHs.

Layer 3 Topology The layer 3 topology used for these experiments can be seen below in . Layer 3 is specifically noted because layer 2 switches exist in the topology as well as some components being virtualized, however, this is not expected to have an effect on the traversal of packets containing EHs.

Both client and server are virtual machines running Ubuntu 22.04 and are configured to inject EHs into all egress packets. Additionally, the server has an out-of-the-box installation of an apache2 web server. Details of the routers used can be found below in .

Routers Under Test (RUTs) Six routers in total were tested and are comprised of three major router manufacturers. Of the six routers, five are physical and the sixth is virtual. Due to confidentiality requirements, the specific manufacturers and models of the routers tested cannot be disclosed, therefore, the routers will be referred to with a numerical value from here on.

Configuration For these experiments, all RUTs have very minimal configurations. No routing protocols are enabled and no ACLs are configured. The routers are configured to simply forward IPv6 traffic between the two interfaces.

Experiments For all experiments, both the client and server are configured to inject varying EHs into all egress packets. A single HTTP request is then made from the client to the server and it was observed that the client received an HTTP reply containing the expected EHs. This indicates that the RUT properly forwarded all packets associated with both the HTTP request and reply that contained EHs without dropping the packets or stripping the EHs from them . To execute the HTTP request, the curl utility was used.

Performance and Diagnostics Metrics (PDM) Destination Option defines the PDM destination option (DO), an option used for collecting performance metrics such as round-trip delay and server delay. For this experiment, a PDM implementation leveraging Extended Berkley Packet Filter (eBPF) was used . The PDM DO was chosen to test with as it is a proposed standard and access to an existing implementation was provided for the use of this research.

Variable Length Hop-by-hop and Destination Options The second type of experiment run was testing how the RUTs processed both HBH and DO of varying header sizes. For both HBH and DO, sizes of 8, 16, 32, 64, 128, 256, and 512 bytes were tested. In addition to this, tests with an EH chain of both HBH and DO for all enumerations of the 8, 16, 32, 64, 126, and 256 header sizes were also run. Overall, there were 48 unique scenarios tested. To achieve this, the IPv6 Extension Headers Injection with eBPF project was used which allows for injecting many different types of extension headers with control over the size of each header.

Results and Findings The following section reports on the findings of these experiments. Tables in this section will denote whether the router properly forwarded packets containing the EH(s) or not. Scenarios in which the router forwards the packets will be denoted with a check mark (✓) and scenarios where the router does not forward the packets will be denoted with an X (X).

Performance and Diagnostics Metrics (PDM) Destination Option As can be seen in , all six of the routers tested properly forwarded packets containing the PDM DO. Router 1 Router 2 Router 3 Router 4 Router 5 Router 6 ✓ ✓ ✓ ✓ ✓ ✓

Variable Length Hop-by-hop and Destination Options

Single Extension Header shows the results of a single variable sized HBH or DO EH. Seen below, all but one router forwards the EHs in every scenario. Scenario Router 1 Router 2 Router 3 Router 4 Router 5 Router 6 HBH 8 ✓ ✓ ✓ ✓ ✓ ✓ HBH 16 ✓ ✓ ✓ ✓ ✓ ✓ HBH 32 ✓ ✓ ✓ ✓ ✓ ✓ HBH 64 ✓ ✓ ✓ ✓ ✓ ✓ HBH 128 ✓ ✓ ✓ ✓ ✓ ✓ HBH 256 ✓ ✓ ✓ ✓ X ✓ HBH 512 ? ? ✓ ? ? ? DO 8 ✓ ✓ ✓ ✓ ✓ ✓ DO 16 ✓ ✓ ✓ ✓ ✓ ✓ DO 32 ✓ ✓ ✓ ✓ ✓ ✓ DO 64 ✓ ✓ ✓ ✓ ✓ ✓ DO 128 ✓ ✓ ✓ ✓ ✓ ✓ DO 256 ✓ ✓ ✓ ✓ X ✓ DO 512 ✓ ✓ ✓ ✓ X ✓ There are two observations to note regarding these results. The first is that five of the six routers have a question mark (?) as their result for the 512-byte HBH scenario. This is because this scenario was unable to be tested for these five routers. The reasoning for this can be found in . The other observation to make is that out of the six routers, Router 5 was the only router unable to forward packets for each scenario. Based on these results, it suggests that Router 5 has a limitation on the maximum size of an EH in a packet. If a packet contains an EH larger than this ceiling, it is dropped. In fact, further testing has shown that a 176-byte HBH or DO is the largest Router 5 is able to properly forward. When increasing to 184-bytes, the initial TCP SYN packet is dropped and therefore the TCP connection is not established and the HTTP request cannot be made.

Multiple Extension Headers The following section includes a matrix for each RUT indicating whether or not it forwarded the HTTP traffic when the packets include both a HBH and DO of varying lengths. DO HBH 8 16 32 64 128 256 8   ✓ ✓ ✓ ✓ ✓ ✓ 16   ✓ ✓ ✓ ✓ ✓ ✓ 32   ✓ ✓ ✓ ✓ ✓ ✓ 64   ✓ ✓ ✓ ✓ ✓ ✓ 128   ✓ ✓ ✓ ✓ ✓ ✓ 256   ✓ ✓ ✓ ✓ ✓ ✓ DO HBH 8 16 32 64 128 256 8   ✓ ✓ ✓ ✓ ✓ ✓ 16   ✓ ✓ ✓ ✓ ✓ ✓ 32   ✓ ✓ ✓ ✓ ✓ ✓ 64   ✓ ✓ ✓ ✓ ✓ ✓ 128   ✓ ✓ ✓ ✓ ✓ ✓ 256   ✓ ✓ ✓ ✓ ✓ ✓ DO HBH 8 16 32 64 128 256 8   ✓ ✓ ✓ ✓ ✓ ✓ 16   ✓ ✓ ✓ ✓ ✓ ✓ 32   ✓ ✓ ✓ ✓ ✓ ✓ 64   ✓ ✓ ✓ ✓ ✓ ✓ 128   ✓ ✓ ✓ ✓ ✓ ✓ 256   ✓ ✓ ✓ ✓ ✓ ✓ DO HBH 8 16 32 64 128 256 8   ✓ ✓ ✓ ✓ ✓ ✓ 16   ✓ ✓ ✓ ✓ ✓ ✓ 32   ✓ ✓ ✓ ✓ ✓ ✓ 64   ✓ ✓ ✓ ✓ ✓ ✓ 128   ✓ ✓ ✓ ✓ ✓ ✓ 256   ✓ ✓ ✓ ✓ ✓ ✓ DO HBH 8 16 32 64 128 256 8   ✓ ✓ ✓ ✓ ✓ X 16   ✓ ✓ ✓ ✓ ✓ X 32   ✓ ✓ ✓ ✓ ✓ X 64   ✓ ✓ ✓ ✓ X X 128   ✓ ✓ ✓ X X X 256   X X X X X X DO HBH 8 16 32 64 128 256 8   ✓ ✓ ✓ ✓ ✓ ✓ 16   ✓ ✓ ✓ ✓ ✓ ✓ 32   ✓ ✓ ✓ ✓ ✓ ✓ 64   ✓ ✓ ✓ ✓ ✓ ✓ 128   ✓ ✓ ✓ ✓ ✓ ✓ 256   ✓ ✓ ✓ ✓ ✓ ✓ Of the six routers, Router 5 () was the only router that did not properly forward packets for all scenarios. As was seen in , Router 5 did not forward packets for the 256-byte and 512-byte scenarios which suggests that Router 5 only processes packets containing EHs up to a certain length. However, in conjunction with these results, it seems as if Router 5 may have limitations based on the total EH chain length rather than individual EHs. It does however properly process smaller EH configurations.

Notable Findings In these results, there are two findings to note. The first is that as seen in and , a router was discovered (i.e., Router 5) that dropped packets in certain scenarios. However, the exact cause of this is still unknown as additional testing not described in this document has shown that whether or not packets are dropped is more nuanced than simply just being based on the length of the EH chain. The second notable finding is that as seen in , the 512-byte HBH scenario was unable to be performed for five out of the six routers. The exact cause is still being investigated, however, packets specifically with a 512-byte HBH option were dropped by the layer 2 switching infrastructure. Notably, both 504-byte and 520-byte HBH options traversed the infrastructure without a problem, and only 512-byte HBH options were affected. Although not recorded in the tables above, both 504-byte and 520-byte HBH scenarios were tested on Routers 1, 2, 3, 4, and 6 and they all forwarded the traffic as expected. Because of this, it is believed this is due to a firmware bug in one of the switches and was an unexpected finding. Router 3 was the only router not affected by this due to it having a different layer 2 path and not traversing the problematic switch.

Conclusion Even though this research was conducted on a relatively small sample size of manufacturers and routers, the results indicate that router implementations in general are likely not the culprits for dropping packets containing HBH or DO EHs. One router did face limitations when processing larger EHs, however, EH chains of that size may not be practical when used in a real-world network and therefore may not be a concern. The most surprising finding was that a layer 2 switch was the cause of dropped packets containing 512-byte HBH options.

Future Work

Determine Causes of Failed Scenarios Currently it is not known what the limitation in Router 5 is caused by and it is also not known why a switch in the layer 2 infrastructure is dropping packets specifically with 512-byte HBH options. Next steps will include trying to determine why these behaviors are occurring and making the manufacturers aware of these limitations with the hopes that they address them if possible.

Additional Routers A logical next step would be to test more routers, especially across a wider breadth of manufacturers. Additionally, more testing could be performed with varying connectors and link speeds.

Additional Extension Headers and Chains This research investigated how routers handle varying sizes of single HBH and DO EHs as well as chains of the two. However, more tests could be run with other standalone EHs as well as in various chain configurations.

Testing Under Load As was mentioned in , these experiments were run with a single HTTP request and therefore were not run under load which may be the case for routers on the open internet. It is possible that a router's EH processing varies based on whether it is under load or not, especially in the context of processing packets on the fast vs slow path. Because of this, running these same experiments at higher throughputs may be interesting to explore.

Security Considerations This document has no security considerations.

Privacy Considerations This document has no privacy considerations.

IANA Considerations This document has no IANA actions.

This document specifies version 6 of the Internet Protocol (IPv6). It obsoletes RFC 2460. To assess performance problems, this document describes optional headers embedded in each packet that provide sequence numbers and timing information as a basis for measurements. Such measurements may be interpreted in real time or after the fact. This document specifies the Performance and Diagnostic Metrics (PDM) Destination Options header. The field limits, calculations, and usage in measurement of PDM are included in this document. Inside Products NITK Surathkal NITK Surathkal NITK Surathkal NITK Surathkal RFC8250 describes an optional Destination Option (DO) header embedded in each packet to provide sequence numbers and timing information as a basis for measurements. As kernel implementation can be complex and time-consuming, this document describes the implementation of the Performance and Diagnostic Metrics (PDM) extension header using eBPF in the Linux kernel's Traffic Control (TC) subsystem. The document also provides a performance analysis of the eBPF implementation in comparison to the traditional kernel implementation. SiPanda This document specifies an experimental method to allow routers to remove IPv6 Hop-by-Hop Options or Routing headers from packets in- flight. The goal is to reduce the probability of packets being dropped because they contain extension headers, without adversely impacting functionality. An additional goal is to limit visibility of information in extension headers to those nodes that need to process the headers.

Acknowledgments The author would like to acknowledge and thank the following individuals. Nalini Elkins and Michael Ackermann for their guidance and support throughout this research. Balajinaidu V, Amogh Umesh, and Chinmaya Sharma for developing and providing access to their PDM DO implementation. Justin Iurman for developing and providing access to their IPv6 Extension Headers Injection with eBPF project. Michayla Newcombe and Ben Patton for their review and feedback of this document.