Operational Issues with Processing of the Hop-by-Hop Options Header

Due to historical reasons, such as limited Application Specific Integrated Circuits (ASICs) support, limited IPv6 deployment, and few service requirements, the most common Hop-by-Hop Options header (HBH) processing implementation is that the node sends the IPv6 packets with the HBH Options Header to the control plane of the node. The option type of each option carried within the HBH Options Header will not even be examined before the packet is sent to the control plane . Very often, such processing behavior is the default configuration or, even worse, is the only behavior of the IPv6 implementation of the node. Sending the HBH Options Header to the control plane for processing could result in Denial of Service (DoS) attack on the router control plane and inconsistent packet drops due to rate-limiting on the interface between the router control plane and forwarding plane, which will either impact the normal end-to-end IP forwarding of the network services. This has in turn led to configuration of measures (such as infrastructure access control lists) to prevent mitigate this DoS vector. This actually introduced a circular problem: -> An implementation problem caused the HBH Options Header to become a DoS vector. -> Because a HBH Options Header is a DoS vector, network operators deployed ACLs that discard packets containing HBH. -> Because network operators deployed ACLs that discard packets containing HBH, network designers stopped defining new HBH Options. -> Because network designers stopped defining new HBH Options, the community was not motivated to fix the implementation problem that caused use of a HBH Option Header to become a DoS vector. Driven by the wide deployments of IPv6 and ever-emerging new services, the HBH Options Header is a valuable container for carrying the information to facilitate these new services. The purpose of this work is to: Break the endless cycle that resulted in HBH being a DOS vector. Enable the HBH options header to be utilized in a safe and secure way without impacting the control plane. Ease the deployments of the new network services that utilize the HBH Option Header in a multi-vendor scenario that can now be deployed without operational impact. In this document, reasons why the HBH is rarely used within networks will be documented, which motivates a list of requirements to allow a better leverage of the HBH capability.

The terms Forwarding Plane and Control Plane used in this draft refer to the terminologies as defined in .

The architecture of modern routers deployed by Internet operators maintains a strict separation of the router control plane and its forwarding plane , as shown in Figure 1. Either the control plane or the forwarding plane is composed of both software and hardware, but each plane is responsible for different functions. In this document, we focus on only the routers following the architecture as shown in Figure 1 and those being deployed in the network rather than those at home which are examples of software-based router.

The router control plane supports control and management functions, handling packets destined to the device as well as building and sending packets originated locally on the device, and also drives the programming of the forwarding plane. The router control plane is generally realized in software on general-purpose processors, and its hardware is usually not optimized for high-speed packet handling. It is more susceptible to security vulnerabilities and a more likely a target for a DoS attack. The forwarding plane is typically responsible for receiving a packet on an incoming interface, performing a lookup to identify the packet's next hop and determine the outgoing interface towards the destination, and forwarding the packet out through the appropriate outgoing interface. Typically, forwarding plane functionality is realized in high-performance ASICs or Network Processors (NPs) that are capable of handling very high packet rates. The router control plane interfaces with its forwarding plane through the Interface Z, as shown in the Figure 1, and the forwarding plane connects to other network devices via Interfaces such as X and Y. Since the router control plane is vulnerable to the DoS attack, usually a traffic filtering mechanism is implemented on Interface Z in order to block unwanted traffic being punted from the forwarding plane to the control plane. To protect the router control plane, a rate-limiting mechanism is always implemented on this interface. However, such a rate limiting mechanism will always cause inconsistent packet drops, which will impact the normal IP forwarding. Many routers allow an ACL rule to be applied to this interface. Semiconductor chip technology has advanced significantly in the last decade, and as such the widely used network processing and forwarding process can now not only forward packets at line speed, but also easily support other feature processing such as QoS for DiffServ/ MPLS, Access Control List (ACL), Firewall, and Deep Packet Inspection (DPI). A Network Processing Unit (NPU) is a non-ASIC based Integrated Circuit (IC) that is programmable through software. It can perform all packet header operations between the physical layer interface and the switching fabric such as packet parsing and forwarding, modification, and forwarding. Many equipment vendors implement these functions in fixed function ASICs rather than using "off-the-shelf" NPUs, because of proprietary algorithms. A Classification Co-processor is a specialized processor that can be used to lighten the processing load on an NPU by handling the parsing and classification of incoming packets such as IPv6 extended header HBH options processing. This approach enables network processors to provide general support for processing simple control messages for traffic management, such as signaling for hardware programming, congestion state report, OAM, etc. An industry trend is for intelligent multi-core CPU hardware using modern NPUs for forwarding packets at line rate while still being able to perform other complex tasks such as HBH forwarding options processing without having to punt to the control plane. Many of the packet-processing devices employed in current switch and router designs are fixed-function ASICs that handle common functions. While these devices can be very efficient for the set of functions they are designed for, they can be very inflexible. There is a tradeoff of price, performance and flexibility when vendors make a choice to use a fixed function ASIC as opposed to NPU. Due to the inflexibility of the fixed function ASIC, tasks that require additional processing such as IPv6 HBH header processing must be punted to the control plane. This problem is still a challenge today and is the reason why operators to protect against control plane DOS attack vector must drop or ignore HBH options. As industry shifts to Merchant Silicon based NPU evolution from fixed function ASIC, the gap will continue to close increasing the viability ubiquitous HBH use cases due to now processing in the forwarding plane. Many routers currently deployed by operators maintain a strict separation between forwarding plane and control plane hardware. Forwarding plane bandwidth and resources are plentiful, while control plane bandwidth and resources are constrained. In order to protect scarce control plane resources, routers enforce policies that restrict access from the forwarding plane to the control plane. Effective policies address packets containing the HBH Options Extension header, because HBH control options require access from the forwarding plane to the control plane. Some operators have considered this as a breach of the separation between the forwarding and control planes. In this case HBH control options would be required to be punted to control plane by fixed function ASICs as well as NPUs. The maximum length of an HBH Options header is 2,048 bytes. The IPv6 standard does not further limit the header chain length or number of options that can be encoded. A source node is permitted to encode hundreds of options in 2,048 bytes . With today's technology it would be cost prohibitive to be able to process hundreds of options with either NPU or proprietary fixed function ASIC . As per , "it is now expected that nodes along a packet's delivery path only examine and process the Hop-by-Hop Options header if explicitly configured to do so". This can be beneficial in cases where transit nodes are legacy hardware and the destination endpoint PE is newer NPU based hardware that can process HBH in the forwarding plane. IPv6 Extended Header limitations that need to be addressed to make HBH processing more efficient and viable in the forwarding plane: defines the IPv6 node requirements and how to protect a node from excessive header chain and excessive header options with various limitations that can be defined on a node. defines ICMPv6 Errors for discarding packets due to processing limits. Per HBH options must be processed serially. However, an implementation of options processing can be done with more parallelism in serial processing by grouping similar options to be processed in parallel. Each Option is encoded in a TLV and so processing of a long list of TLVs is expensive. Zero data length encoded options TLVs are a valid option. A DOS vector could be easily generated by encoding 1000 HBH options (Zero data length) in a standard 1500 MTU packet. Restricting the number of options could avoid an arbitrary large amount of processing for a single EH.

defines several IPv6 extension header types, including the HBH Options header. As specified in , the HBH Options header is used to carry optional information that will be examined and processed by every node along a packet's delivery path, and it is identified by a Next Header value of zero in the IPv6 header. The HBH Options Header contains the following fields: -- Next Header: 8-bit selector, identifies the type of header immediately following the Hop-by-Hop Options header. -- Hdr Ext Len: 8-bit unsigned integer, the length of the Hop-by-Hop Options header in 8-octet units, not including the first 8 octets. -- Options: Variable-length field, of length such that the complete Hop-by-Hop Options header is an integer multiple of 8 octets long. The HBH Options Header as well as the Destination Options Header carries a variable number of "options" that are encoded in the format of type-length-value (TLV). The highest-order two bits (i.e., the ACT bits) of the Option Type specify the action that must be taken if the processing IPv6 node does not recognize the Option Type. This specification is updated in . The third-highest-order bit (i.e., the CHG bit) of the Option Type specifies whether or not the Option Data of that option can change en route to the packet's final destination. As per , it is now expected that nodes along a packet's delivery path only examine and process the Hop-by-Hop Options header if explicitly configured to do so. This means that the HBH processing behavior in a node depends on its configuration. However, in the current , there is no explicit specification of the possible configurations. Therefore, the nodes may be configured to ignore the HBH Options Header, drop packets containing a HBH Options Header, or assign packets containing a HBH Options Header to the control plane . Because of these likely uncertain processing behaviors, new hop-by-hop options are not recommended. This specification is updated in .

Current routers deployed by operators usually process an IPv6 packet that has a Next Header field set to 0 by directly forwarding it to the control plane of the node. With such implementations, the value of the Next Header field in the IPv6 header is the only trigger for the default processing behavior. The option type of each option carried within the HBH Options Header will not even be examined before the packet is sent to the control plane. Very often, such processing behavior is the default configuration on the node, which is embedded in the implementation and cannot be changed or reconfigured. Another critical component of IPv6 HBH processing, in some cases overlooked, is that an operator core network can be designed to use the global Internet routing table for internet traffic and in other cases use an overlay MPLS VPN to carry Internet traffic. In the global Internet routing table scenario where only an underlay global routing table exists, and no VPN overlay carrying customer Internet traffic, the IPv6 HBH options could be used as a DOS attack vector for both the operator nodes, adjacent inter-AS peer nodes as well as customer nodes along a path. In a case where the Internet routing table is carried in a MPLS VPN overlay payload, the HBH options header does not impact the operator underlay framework and only impacts the VPN overlay payload and thus the operator underlay top most label global table routing FEC LSP instantiation is not impacted as the operator underlay is within the operators closed domain. However, the HBH options DOS attack vector in the VPN overlay can still impact the customer CE destination end nodes as well as other adjacent inter-AS operators that only use the underlay global Internet routing table. In an operator domain where MPLS VPN overlay is utilized to carry internet traffic, the operator has full control of the underlay and IPv6 Extension Header chain length as well as the number of HBH options encoded . In the global routing table scenario for Internet traffic there is no way to control the IPv6 Extended header chain length as well as the number of HBH options encoded.

When IPv6 was first implemented on high-speed routers, HBH options were not yet well-understood and ASICs were not as capable as they are today. So, early IPv6 implementations dispatched all packets that contain HBH options to their control plane.

Such implementation introduces a risk of a DoS attack on the control plane of the node, and a large flow of IPv6 packets could congest the control plane, causing other critical functions (including routing and network management) that are executed on the control plane to fail. Rate limiting mechanisms will cause inconsistent packet drops and impact the normal end-to-end IP forwarding of the network services.

To mitigate this DoS vulnerability, many operators have deployed Access Control Lists (ACLs) that discard all packets containing HBH Options . shows the Reports from the field indicating that some IP routers deployed within the global Internet are configured either to ignore or to drop packets having a hop-by-hop header. As stated in , many network operators perceive HBH Options to be a breach of the separation between the forwarding and control planes. Therefore, several network operators configured their nodes so as to discard all packets containing the HBH Options Extension Header, while others configured nodes to forward the packet but to ignore the HBH Options. also states that hop-by-hop options are not handled by many high-speed routers or are processed only on a control plane. shows that the HBH options header cannot reliably traverse the global Internet; only small headers with 'skippable' options have some chances. Due to such behaviors observed and described in these specifications, did not recommend new HBH Options, which limited the usability of HBH Options.. Besides service providers' networks, other sectors such as industrial IoT networks are slowly replacing a dozen of semi-proprietary protocols in industrial automation into IP. The proper processing of the HBH options header is also required.

As IPv6 is being rapidly and widely deployed worldwide, more and more applications and network services are migrating to or directly adopting IPv6. More and more new services that require HBH are emerging and the HBH Options header is going to be utilized by the new services in various scenarios. In-situ OAM (IOAM) with IPv6 encapsulation is one of the examples. IOAM in IPv6 is used to enhance diagnostics of IPv6 networks and complements other mechanisms, such as the IPv6 Performance and Diagnostic Metrics Destination Option described in . The IOAM data fields are encapsulated in "option data" fields of the Hop-by-Hop Options header. The Alternate Marking Method can be used as the passive performance measurement tool in an IPv6 domain. The AltMark Option is defined as a new IPv6 extension header option to encode alternate marking technique and Hop-by-Hop Options Header is considered . The Minimum Path MTU Hop-by-Hop Option is defined in to record the minimum Path MTU along the forward path between a source host to a destination host. This Hop-by-Hop option is intended to be used in environments like Data Centers and on paths between Data Centers as well as other environments including the general Internet. It provides a useful tool to better take advantage of paths capable of supporting a large Path MTU. A Virtual Transport Network (VTN) is a virtual underlay network which consists of a set of dedicated or shared network resources allocated from the physical underlay network, and is associated with a customized logical network topology. The VTN Option is used to carry the VTN related information, which could used to identify the set of network resources allocated to a VTN and the rules for packet processing. . As more services start utilizing the HBH Options header, more packets containing HBH Options are going to be injected into the networks. A currently common configuration in deployed networks is for routers to send all the packets of the new services to the control plane of the nodes, with the possible consequence of resulting in a DoS vulnerability on the control plane. The packets will be dropped and the normal IP forwarding may be severely impacted. The deployment of new network services involving multi-vendor interoperability will become impossible.

The HBH Options Header should not become a possible DDoS Vector. Therefore, the control plane needs to be preserved from unwanted incoming traffic due to HBH header present in the packet. HBH Options need to be designed in a manner so that they don't reduce the probability of packet delivery. HBH processing needs to reduce the forwarding rate. Implementations need to perform well at a reasonable cost without endanger the security of the router. The Router Alert Option needs not to impact the processing of other HBH Options that should be processed more quickly. HBH Options can influence how a packet is forwarded. However, with the exception of the Router Alert Option, an HBH Option MUST NOT cause control plane state to be created, modified or destroyed on the processing node. As per , protocol developers SHOULD avoid future use of the Router Alert Option.

In order to make the HBH Options Header usable and facilitate the ever-emerging new services to be deployed across multiple vendors' devices, the new specifications related to the processing of HBH Options Header, needs to be designed to allow a smooth migration from old to new behavior without disruption time.

The security considerations can refer to , , and .

This document does not include an IANA request.

The authors would like to acknowledge Ron Bonica, Fred Baker, Bob Hinden, Stefano Previdi, and Donald Eastlake, Gorry Fairhurst for their valuable review and comments.