An Information Model for Packet Discard Reporting

An Information Model for Packet Discard Reporting Amazon

1 Principal Place, Worship Street London EC2A 2FA UK jevanamz@amazon.co.uk

Amazon

410 Terry Ave N Seattle WA 98109 US opyl@amazon.com

Juniper Networks

1133 Innovation Way Sunnyvale CA 94089 US jhaas@juniper.net

Cisco Systems, Inc.

170 West Tasman Dr. San Jose CA 95134 US akadosh@cisco.com

Operations and Management Area OPSAWG Internet-Draft The primary function of a network is to transport packets and deliver them according to a service level objective. Understanding both where and why packet loss occurs within a network is essential for effective network operation. Device-reported packet loss is the most direct signal for network operations to identify customer impact resulting from unintended packet loss. This document defines an information model for packet loss reporting, which classifies these signals to enable automated network mitigation of unintended packet loss.

Introduction In automating network operations, a network operator needs to be able to detect anomalous packet loss, diagnose or root cause the loss, and then apply one of a set of possible actions to mitigate customer-impacting packet loss. Some packet loss is normal or intended in IP networks, however. Hence, precise classification of packet loss signals is crucial both to ensure that anomalous packet loss is easily detected and that the right action or sequence of actions are taken to mitigate the impact, as taking the wrong action can make problems worse. The existing metrics for reporting packet loss, as defined in - namely ifInDiscards, ifOutDiscards, ifInErrors, ifOutErrors - do not provide sufficient precision to automatically identify the cause of the loss and mitigate the impact. From a network operator's perspective, ifInDiscards can represent both intended packet loss (e.g., packets discarded due to policy) and unintended packet loss (e.g., packets dropped in error). Furthermore, these definitions are ambiguous, as vendors can and have implemented them differently. In some implementations, ifInErrors accounts only for errored packets that are dropped, while in others, it accounts for all errored packets, whether they are dropped or not. Many implementations support more discard metrics than these; where they do, they have been inconsistently implemented due to the lack of a standardised classification scheme and clear semantics for packet loss reporting. provides support for reporting discards per flow in IPFIX using forwardingStatus, however, the defined drop reason codes also lack sufficient clarity to support automated root cause analysis and mitigation of impact. Hence, this document defines an information model for packet loss reporting, aiming to address these issues by presenting a packet loss classification scheme that can enable automated mitigation of unintended packet loss. Consistent with , this information model is independent of any specific implementations or protocols used to transport the data. There are multiple ways that this information model could be implemented (i.e., data models), including SNMP , NETCONF / YANG , and IPFIX , but they are outside of the scope of this document. We further limit the scope of this document to reporting packet loss at layer 3 and frames discarded at layer 2, although the information model could be extended in future to cover segments dropped at layer 4. Section 3 describes the problem. Section 4 defines the information model and semantics with examples. Section 5 provides examples of discard signal-to-cause-to-auto-mitigation action mapping. Appendix B details the authors' experience from implementing this model. This document considers only the signals that may trigger automated mitigation plans and not how they are defined or executed.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. A packet discard is considered to be any packet dropped by a device, which may be intentional (i.e. due to a configured policy, e.g. such as an Access Control List (ACL)) or unintentional (i.e. packets dropped in error).

Problem Statement At the highest-level, unintended packet loss is the discarding of packets that the network operator otherwise intends to deliver, i.e. which indicates an error state. There are many possible reasons for unintended packet loss, including: erroring links may corrupt packets in transit; incorrect routing tables may result in packets being dropped because they do not match a valid route; configuration errors may result in a valid packet incorrectly matching an access control list (ACL) and being dropped. Whilst the specific definition of unintended packet loss is network dependent, for any network there are a small set of potential actions that can be taken to minimise customer impact by auto-mitigating unintended packet loss: Take a device, link, or set of devices and/or links out of service. Return a device, link, or set of devices and/or links back into service. Move traffic to other links or devices. Roll back a recent change to a device that might have caused the problem. Escalate to a human (e.g., network operator) as a last resort. A precise signal of impact is crucial, as taking the wrong action can be worse than taking no action. For example, taking a congested device out of service can make congestion worse by moving the traffic to other links or devices, which are already congested. To detect whether device-reported discards indicate a problem and to determine what actions should be taken to mitigate the impact and remediate the cause, depends on four primary features of the packet loss signal: The cause of the loss. The rate and/or degree of the loss. The duration of the loss. The location of the loss. Features 2, 3, and 4 are already addressed with passive monitoring statistics, for example, obtained with SNMP / MIB-II or NETCONF / YANG . Feature 1, however, is dependent on the classification scheme used for packet loss reporting. In the next section, we define a new classification scheme to address this problem.

Information Model The classification scheme is defined as a tree which follows the structure component/direction/type/layer/sub-type/sub-sub-type/.../metric, where:
a. component can be interface|device|control_plane|flow
b. direction can be ingress|egress
c. type can be traffic|discards, where traffic accounts for packets successfully received or transmitted, and discards accounts for packet drops
d. layer can be l2|l3

For additional context, Appendix A provides an example of where packets may be discarded in a device.

Discard Class Descriptions

discards/policy/:: These are intended discards, meaning packets dropped by a device due to a configured policy. There are multiple sub-classes.
discards/error/l2/rx/:: Frames discarded due to errors in the received L2 frame. There are multiple sub-classes, such as those resulting from failing CRC, invalid header, invalid MAC address, or invalid VLAN.
discards/error/l3/rx/:: These are discards which occur due to errors in the received packet, indicating an upstream problem rather than an issue with the device dropping the errored packets. There are multiple sub-classes, including header checksum errors, MTU exceeded, and invalid packet, i.e. due to incorrect version, incorrect header length, or invalid options.
discards/error/l3/rx/ttl_expired:: There can be multiple causes for TTL-expired (or Hop limit exceeded) discards: i) trace-route; ii) TTL (Hop limit) set too low by the end-system; iii) routing loops.
discards/error/l3/no_route/:: Discards occur due to a packet not matching any route.
discards/error/local/:: A device may discard packets within its switching pipeline due to internal errors, such as parity errors. Any errored discards not explicitly assigned to the above classes are also accounted for here.
discards/no_buffer/:: Discards occur due to no available buffer to enqueue the packet. These can be tail-drop discards or due to an active queue management algorithm, such as RED or CODEL .

Requirements Requirements 1-10 relate to packets forwarded by the device; requirement 11 relates to packets destined to or from the device: All instances of frame or packet receipt, transmission, and discards MUST be reported. All instances of frame or packet receipt, transmission, and discards SHOULD be attributed to the physical or logical interface of the device where they occur. An individual frame MUST only be accounted for by either the L2 traffic class or the L2 discard classes within a single direction, i.e., ingress or egress. An individual packet MUST only be accounted for by either the L3 traffic class or the L3 discard classes within a single direction, i.e., ingress or egress. A frame accounted for at L2 SHOULD NOT be accounted for at L3 and vice versa. An implementation MUST expose which layers a discard is counted against. The aggregate L2 and L3 traffic and discard classes SHOULD account for all underlying packets received, transmitted, and discarded across all other classes. The aggregate Quality of Service (QoS) traffic and no buffer discard classes MUST account for all underlying packets received, transmitted, and discarded across all other classes. In addition to the L2 and L3 aggregate classes, an individual discarded packet MUST only account against a single error, policy, or no_buffer discard subclass. When there are multiple reasons for discarding a packet, the ordering of discard class reporting MUST be defined. If Diffserv is not used, no_buffer discards SHOULD be reported as class0. Traffic to the device control plane has its own class, however, traffic from the device control plane SHOULD be accounted for in the same way as other egress traffic.

Examples Assuming all the requirements are met, a "good" unicast IPv4 packet received would increment:
- interface/ingress/traffic/l3/v4/unicast/packets
- interface/ingress/traffic/l3/v4/unicast/bytes
- interface/ingress/traffic/qos/class_0/packets
- interface/ingress/traffic/qos/class_0/bytes A received unicast IPv6 packet discarded due to Hop Limit expiry would increment:
- interface/ingress/discards/l3/v6/unicast/packets
- interface/ingress/discards/l3/v6/unicast/bytes
- interface/ingress/discards/l3/rx/ttl_expired/packets An IPv4 packet discarded on egress due to no buffers would increment:
- interface/egress/discards/l3/v4/unicast/packets
- interface/egress/discards/l3/v4/unicast/bytes
- interface/egress/discards/no_buffer/class_0/packets
- interface/egress/discards/no_buffer/class_0/bytes

Example Signal-Cause-Mitigation Mapping gives an example discard signal-to-cause-to-mitigation action mapping. Mappings for a specific network will be dependent on the definition of unintended packet loss for that network.

Security Considerations There are no new security considerations introduced by this document.

IANA Considerations There are no new IANA considerations introduced by this document.

Contributors

Acknowledgments The content of this draft has benefitted from feedback from JR Rivers, Ronan Waide, Chris DeBruin, and Marcoz Sanz.

Key words for use in RFCs to Indicate Requirement Levels In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Management Information Base for Network Management of TCP/IP-based internets: MIB-II This memo defines the second version of the Management Information Base (MIB-II) for use with network management protocols in TCP/IP-based internets. [STANDARDS-TRACK] Simple Network Management Protocol (SNMP) This RFC is a re-release of RFC 1098, with a changed "Status of this Memo" section plus a few minor typographical corrections. This memo defines a simple protocol by which management information for a network element may be inspected or altered by logically remote users. [STANDARDS-TRACK] On the Difference between Information Models and Data Models There has been ongoing confusion about the differences between Information Models and Data Models for defining managed objects in network management. This document explains the differences between these terms by analyzing how existing network management model specifications (from the IETF and other bodies such as the International Telecommunication Union (ITU) or the Distributed Management Task Force (DMTF)) fit into the universe of Information Models and Data Models. This memo documents the main results of the 8th workshop of the Network Management Research Group (NMRG) of the Internet Research Task Force (IRTF) hosted by the University of Texas at Austin. This memo provides information for the Internet community. IP Flow Information Export (IPFIX) Implementation Guidelines The IP Flow Information Export (IPFIX) protocol defines how IP Flow information can be exported from routers, measurement probes, or other devices. This document provides guidelines for the implementation and use of the IPFIX protocol. Several sets of guidelines address Template management, transport-specific issues, implementation of Exporting and Collecting Processes, and IPFIX implementation on middleboxes (such as firewalls, network address translators, tunnel endpoints, packet classifiers, etc.). This memo provides information for the Internet community. Network Configuration Protocol (NETCONF) The Network Configuration Protocol (NETCONF) defined in this document provides mechanisms to install, manipulate, and delete the configuration of network devices. It uses an Extensible Markup Language (XML)-based data encoding for the configuration data as well as the protocol messages. The NETCONF protocol operations are realized as remote procedure calls (RPCs). This document obsoletes RFC 4741. [STANDARDS-TRACK] The YANG 1.1 Data Modeling Language YANG is a data modeling language used to model configuration data, state data, Remote Procedure Calls, and notifications for network management protocols. This document describes the syntax and semantics of version 1.1 of the YANG language. YANG version 1.1 is a maintenance release of the YANG language, addressing ambiguities and defects in the original specification. There are a small number of backward incompatibilities from YANG version 1. This document also specifies the YANG mappings to the Network Configuration Protocol (NETCONF). Random Early Detection gateways for Congestion Avoidance An Architecture for Differentiated Services This document defines an architecture for implementing scalable service differentiation in the Internet. This memo provides information for the Internet community. Controlled Delay Active Queue Management This document describes CoDel (Controlled Delay) -- a general framework that controls bufferbloat-generated excess delay in modern networking environments. CoDel consists of an estimator, a setpoint, and a control loop. It requires no configuration in normal Internet deployments. Cisco-Specific Information Elements Reused in IP Flow Information Export (IPFIX) This document describes some additional IP Flow Information Export (IPFIX) Information Elements in the range of 1-127, which is the range compatible with field types used by NetFlow version 9 in RFC 3954, as specified in the IPFIX Information Model in RFC 7012.

Where do packets get dropped? depicts an example of where and why packets may be discarded in a typical single ASIC, shared buffered type device, where packets ingress on the left and egress on the right.

Phy +--> Mac +--> Ingress +--> Buffers +--> Egresss +--> Mac +--> Phy |> Packet tx | | | | | Pipeline| | | | Pipeline| | | | | +----------+ +----------+ +----------+ +----------+ +----------+ +----------+ +----------+ Intended policy/acl policy/acl Discards: policy/policer policy/policer policy/urpf null_route Unintended error/rx/l2 error/rx/l3 no_buffer error/tx/l3 Discards: error/local no_route ttl ]]>

Implementation Experience This appendix captures the authors' experience gained from implementing and applying this information model across multiple vendors' platforms, as guidance for future implementers. The number and granularity of classes described in Section 3 represent a compromise. It aims to offer sufficient detail to enable appropriate automated actions while avoiding excessive detail, which may hinder quick problem identification. Additionally, it helps constrain the quantity of data produced per interface to constrain data volume and device CPU impacts. Although further granularity is possible, the scheme described has generally proven to be sufficient for the task of auto-mitigating unintended packet loss. There are multiple possible ways to define the discard classification tree. For example, we could have used a multi-rooted tree, rooted in each protocol. Instead, we opted to define a tree where protocol discards and causal discards are accounted for orthogonally. This decision reduces the number of combinations of classes and has proven sufficient for determining mitigation actions. NoBuffer discards can be realized differently with different memory architectures. Whether a NoBuffer discard is attributed to ingress or egress can differ accordingly. For successful auto-mitigation, discards due to egress interface congestion should be reported on egress, while discards due to device-level congestion (e.g. due to exceeding the device forwarding rate) should be reported on ingress. Platforms often account for the number of packets discarded where the TTL has expired (or Hop Limit exceeded), and the device CPU has returned an ICMP Time Exceeded message. There is typically a policer applied to limit the number of packets sent to the device CPU, however, which implicitly limits the rate of TTL discards that are processed. One method to account for all packet discards due to TTL expired, even those that are dropped by a policer when being forwarded to the CPU, is to use accounting of all ingress packets received with TTL=1. Where no route discards are implemented with a default null route, separate discard accounting is required for any explicit null routes configured, in order to differentiate between interface/ingress/discards/policy/null_route/packets and interface/ingress/discards/errors/no_route/packets. It is useful to account separately for transit packets discarded by transit ACLs or policers, and packets discarded by ACLs or policers which limit the number of packets to the device control plane. It is not possible to identify a configuration error - e.g., when intended discards are unintended - with device packet loss metrics alone. For example, to determine if ACL drops are intended or due to a misconfigured ACL some other method is needed, i.e., with configuration validation before deployment or by detecting a significant change in ACL drops after a configuration change compared to before. Where traffic byte counters need to be 64-bit, packet and discard counters that increase at a lower rate may be encoded in fewer bits, e.g., 48-bit. Aggregate counters need to be able to deal with the possibility of discontinuities in the underlying counters. In cases where the reporting device is the source or destination of a tunnel, the ingress protocol for a packet may differ from the egress protocol; if IPv4 is tunneled over IPv6 for example. Some implementations may attribute egress discards to the ingress protocol. While the classification tree is seven layers deep, a minimal implementation may only implement the top six layers.