]> Cisco Systems, Inc.

Canada rgandhi@cisco.com

Ericsson

gregimirsky@gmail.com

Juniper Networks

tony.li@tony.li

Futurewei Technologies

USA haoyu.song@futurewei.com

Comcast

Bin_Wen@cable.comcast.com

Routing MPLS Working Group Internet-Draft IOAM MPLS Network Action In situ Operations, Administration, and Maintenance (IOAM), defined in RFC 9197, is an on-path telemetry method to collect and record the operational state and telemetry information using, for example, Pre-allocated, Proof-of-Transit, Edge-To-Edge or Incremental IOAM Options, that can be used to calculate various performance metrics. RFC 9326 defined the IOAM Direct Export (IOAM-DEX) Option in which the operational state and telemetry information are collected according to the specified profile and exported in a manner and format defined by a local policy on each node along the path. MPLS Network Actions (MNA) techniques are meant to indicate actions to be performed on any combination of Label Switched Paths, MPLS packets, and the node itself, and to transfer data needed for these actions. This document explores the MNA mechanisms to collect and transport the on-path operational state, and telemetry information IOAM data fields, including IOAM-DEX Option.

Introduction In situ OAM (IOAM) is an on-path telemetry method used to collect and record the operational state and telemetry information that can be used to calculate various performance metrics. Several IOAM Option types (e.g., Pre-allocated, Proof-of-Transit, Edge-To-Edge, and Incremental) use the user packet to collect the operational state and telemetry information. Such mechanisms transport the collected information to an IOAM decapsulating node (typically located at the edge of the IOAM domain within the data packet). IOAM Direct Export (IOAM-DEX) is an IOAM Option type. In IOAM-DEX, the operational state and telemetry information are collected according to the specified profile and exported in a manner and format defined by a local policy on each node along the path. MPLS Network Actions (MNA) techniques indicate actions to be performed on any combination of Label Switched Paths, MPLS packets, the node itself, and also allow for the transfer of data needed for these actions. defines mechanisms for carrying the Network Action Sub-Stack (NAS) as part of the MPLS label stack, i.e., the In-Stack MNA solution. defines mechanisms for carrying MNA and Ancillary Data (AD) outside the MPLS label stack, i.e., as the Post-Stack MNA solution. describes various use cases that can be realized using MNA techniques, including IOAM Pre-allocated, Proof-of-Transit, Edge-To-Edge, Incremental, and IOAM-DEX Option types. This document describes how MNA can be used for collecting and transporting on-path operational state and telemetry information using IOAM data fields for IOAM Option types, including IOAM-DEX. Specifying the mechanism for exporting the collected information in case of the IOAM-DEX Option is outside the scope of this document.

Conventions Used in This Document

OAM: Operations, Administration, and Maintenance HBH: Hop-By-Hop I2E: Ingress-To-Egress IHS: Ingress-To-Egress (I2E), Hop-By-Hop (HBH) or Select Scope IOAM: In situ OAM IOAM-DEX: IOAM Direct Export IOAM-DEX-ISD-MNA: IOAM Direct Export as MPLS Network Action ISD ISD: In-Stack Data PFN: Post-Stack First Nibble PSD: Post-Stack Data PSMH: Post-Stack MPLS Header LSE: Label Stack Entry MPLS: Multiprotocol Label Switching MNA: MPLS Network Action NAI: Network Action Indicator NAS: Network Action Sub-stack NASL: Network Action Sub-stack Length

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Applicability of IOAM and IOAM-DEX in an MPLS Network Pre-allocated, Incremental, Proof-of-Transit, and Edge-to-Edge IOAM Option types use user packets to collect and transport the operational state and telemetry information. In some environments, for example, data center networks, this technique is useful as the available bandwidth and the use of jumbo frames can accommodate the increase of the packet payload. This document defines the Post-Stack MNA () solution supporting Pre-allocated, Incremental, Proof-of-Transit, and Edge-to-Edge IOAM Option types (). However, for some use cases, e.g., mobile backhaul, in which network resources are closely controlled, collecting and transporting the telemetry information within a data packet may noticeably decrease the cost-efficiency of network operations. Although the operational state and telemetry information are essential for network automation (Section 4 of ), its delivery is not as critical as user packets. As such, collecting and transporting the operational state and telemetry information using the management plane is a viable option for some environments. IOAM-DEX is capable of collecting the IOAM data fields defined in . The processing and transport of the collected information are controlled by a local policy which is outside the scope of this specification. The performance considerations discussed in Section 5 of are applicable here. In this document, the realizations of IOAM-DEX using the In-Stack MNA ( and Post-Stack MNA () are defined in and , respectively.

Realization of IOAM and IOAM-DEX as MPLS Network Actions

Realization of IOAM and IOAM-DEX as Post-Stack MNA The presence of Post-Stack MPLS Header is indicated in an In-Stack Network Action Sub-stack as defined in and is shown in .

An Example of In-Stack Network Action Sub-Stack with Post-Stack MPLS Header Carrying IOAM and IOAM-DEX

The MNA Label (value TBA4) is defined in . The IHS scope field defined in is used to indicate that I2E or HBH or Select processing is required for the Network Action and Ancillary Data. If both edge and intermediate nodes need to process the IOAM data fields then IHS scope MUST be set to "HBH, value 0x1". If only edge nodes need to process the IOAM data fields then IHS scope MUST be set to "I2E, value 0x0". The HBH scope allows skipping the IOAM data processing on the intermediate nodes i.e., avoids the need to parse all IOAM data fields to detect the HBH option type. The U Flag for Unknown Action Handling is specified in . S bit is the Bottom of Stack field . IHS field is set as specified in . P flag is set as specified in . The Network Action Sub-Stack Length (NASL) is set as specified in . Opcode (7 bits) value is set to TBA1 (for Network Action for IOAM and IOAM-DEX in PSD) for carrying an IOAM Option-Type and the associated data fields defined in and the IOAM-DEX Option-Type and the associated data fields defined in in PSMH. This network action is optional and can be carried in Format B or Format C Label Stack Entry (LSE). Data (10 bits, next to the Opcode field) in LSE contains the offset for PSMH for this In-Stack Network Action in 4-octets units after BOS LSE to the start of the corresponding Post-Stack Network Action Opcode. Due to the Post-Stack MPLS Header type top-header, the minimum value for the offset is 1 (i.e, 4-octets). Length of Network Action (NAL) is set to 0 for the In-Stack Network Action for IOAM and IOAM-DEX option type in Post-Stack ancillary data. A packet may carry more than one In-Stack Network Action in an MNA Sub-Stack for IOAM and IOAM-DEX in PSMH (for example, for different IOAM Option-Types as identified in Post-Stack ancillary data). There may be a different In-Stack Network Action (other than for the IOAM and IOAM-DEX) in the In-Stack MNA Sub-Stack.

MPLS Network Action Post-Stack MPLS Header for IOAM and IOAM-DEX An example encoding for IOAM and IOAM-DEX carried in Post-Stack MPLS Header is shown in .

An Example of Post-Stack MPLS Header Carrying IOAM and IOAM-DEX

The Post-Stack MPLS Header (PSMH) type top-header is added with the following fields as defined in .

PFN:

The Post-Stack first nibble (PFN) (value TBA3) as defined in .

PS-HDR-LEN (8 bits):

PSMH total length in four-octet units. This excludes the PSMH type top-header.

TYPE:

TYPE is defined for MNA-POST-STACK-HDR (value 1) in .

The PSMH contains Post-Stack Network Action Opcode for IOAM and IOAM-DEX, length in number of 4-octet units, and IOAM Option-Type with IOAM data fields in the Post-Stack ancillary data as shown in . The IOAM data fields MUST follow the definitions corresponding to their IOAM-Option-Types (e.g., see Section 4.4 of and IOAM-DEX Option-Type in Section 3 of ). An IOAM option is added in the PSMH containing the following fields:

MNA-PS-OP:

IANA allocated value TBA1 (Network Action Opcode for IOAM and IOAM-DEX in PSD) for IOAM Option-Type defined in , and IOAM-DEX Option-Type defined .

IOAM-OPT-TYPE:

7-bit field defining the IOAM Option-Type, as defined in the "IOAM Option-Type Registry" specified in and ).

PS-NAL:

7-bit unsigned integer. Length of the IOAM data fields in 4-octet units for the MNA-PS-OP. This excludes the first 4-octet unit.

IOAM Option and Data Space:

IOAM data fields as specified by the IOAM-OPT-Type field. IOAM data fields are defined corresponding to the IOAM-Option-Type (e.g., see Section 4.4 of and Section 3 of .

BLOCK-NUMBER:

The block number for the alternate marking method can be used to aggregate the IOAM data collected in the data plane, e.g., to compute measurement metrics for each block of a data flow. It is also used to correlate the IOAM data on different nodes along the packet path. The block number is incremented sequentially at every measurement interval provisioned on the encapsulating node.

Multiple IOAM and IOAM-DEX Option-Types in PSMH An example of multiple Post-Stack network actions with the same scope carrying different IOAM option types is shown in Figure 3.

Example of Multiple Post-Stack Network Actions with IOAM and IOAM-DEX with Same Scope

Realization of IOAM-DEX as In-Stack MNA This document uses all the elements of the IOAM-DEX Option-Type format defined in to support IOAM-DEX in an MPLS network using MNA ISD . To support the direct export of the operational state and telemetry information, the IOAM-DEX-ISD-MNA blob (binary large object) is placed as part of the ISD block in an MPLS label stack according to the MNA encoding principles defined in . Using the IHS field, the IOAM-DEX-ISD-MNA can be performed in Hop-by-Hop, Ingress-to-Egress, or Select modes of collecting the operational state and telemetry information, using an MNA Opcode ().

An Example of IOAM-DEX Encapsulation as an MNA Opcode

Here, the enclosed elements are defined as follows:

• The MNA Label (value TBA4) is defined in .
• S - the Bottom of Stack field .
• P flag is set as specified in .
• IHS, U, and NASL fields are set as specified in .
• NASL - number of LSEs that compose the IOAM-DEX-ISD-MNA blob.
• Opcode is MNA-IOAM-DEX opcode (value TBA2) assigned by IANA.
• IOAM-DEX-ISD-MNA - IOAM-DEX in MPLS Network Action ISD encoding.

Policies controlling the processing of the collected operational state and telemetry information, and its transport are outside the scope of this document.

IOAM-DEX Option-Type Encoding as MNA ISD The IOAM-DEX in MNA header uses the LSE Format D, as defined in Section 4.4 , that maps IOAM-DEX Option Type format . In addition to the requirement to preserve the Bottom of Stack field, the most significant bit in LSE Format D is always set to 1 avoiding a possible mix-up of the LSE with one of the Base Special Purpose Labels. The format of IOAM-DEX-MNA blob in MNA ISD header is shown in .

IOAM-DEX Option Type Format for MPLS Network Action ISD

Where fields are defined as follows:

• Namespace-ID is a 16-bit identifier of the IOAM Namespace, as defined in .
• Resv field is Reserved, it MUST be set to 0 on transmit and ignored on receipt.
• S is a one-bit the Bottom of Stack .
• Flags is an eight-bit field comprised of eight one-bit subfields. The subfields in the Flags field are allocated by IANA in IOAM-DEX Flags registry, as defined in Section 4.2 of .
• IOAM-Trace-Type-MNA is a 22-bit field. The interpretation of bit positions in the IOAM-Trace-Type-MNA are as specified in , from Bit 0 through Bit 21. Note that bits as specified in are right-shifted one bit, so for example, bit 0 in (left most bit for hop_Lim and node_id in short format) is mapped to Bit 1 (second left most bit) in the MNA encoding. The registry that contains the assigned codepoints is found in the IANA's IOAM Trace-Type registry .
• O is a one-bit flag identical to the interpretation of Bit 22 marked as "variable-length Opaque State Snapshot" as defined in and assigned codepoints found in the IANA's IOAM Trace-Type registry .
• R (Reserved) is a one-bit flag identical to the interpretation of Bit 23 marked as "reserved" as defined in and assigned codepoints found in IANA's IOAM Trace-Type registry . Similarly to , the R bit is reserved to allow for future extensions of the IOAM-Trace-Type-MNA bit field, i.e., it indicates the presence of the extended IOAM-Trace-Type-MNA field in the next LSE in Format D.
• The concatenation of the 22-bit IOAM-Trace-Type-MNA, O, and R fields, explained above, is identical to IOAM-Trace-Type in the interpretation of its bits, as defined in and assigned codepoints found in IANA's IOAM Trace-Type registry .
• Ext-Flags is a six-bit field comprised of six one-bit subfields. The allocation of the subfields in the Ext-Flags field is according to Section 4.3 of . The allocated flags indicate the presence of the optional Flow ID MNA and/or Sequence Number MNA fields in the IOAM-DEX-ISD-MNA header. The length of the Ext-Flags field in IOAM-DEX Option-Type in MNA is shorter by two one-bit fields compared to the length of the Extension Flags field defined in Section 3.2 of . Mapping of these two bit positions are for further study. displays the detailed format of the Ext-Flags field.
• Optional fields, i.e., Flow ID MNA and Sequence Number MNA, according to , immediately follow the Reserved field used to align optional fields at the four-octet word boundary. In the case of IOAM-DEX in MNA, such alignment can be achieved without using padding.
• Flow ID MNA is an optional four-octet field and carries a 30-bit Flow ID (after removing leading 1 and S bits). The semantics of the Flow ID MNA field is as of the Flow ID field defined in Section 3.2 of . The most significant bit MUST be set to 1. Bit 23 MUST be set according to the definition of Bottom of Stack field in .
• Sequence Number MNA is an optional four-octet field and carries a 30-bit sequence number (after removing leading 1 and S bits). The semantics of the Sequence Number MNA field is as of the Sequence Number field defined in Section 3.2 of . The most significant bit MUST be set to 1. Bit 23 MUST be set according to the definition of Bottom of Stack field in . In MPLS network environments where a label stack information is used for load-balancing flows, the 19-bit-long part of the Sequence Number MNA, starting from the Bit 1 position of the LSE, MUST remain immutable for a particular packet flow that the value of the Flow ID MNA field identifies. In MPLS networks, where other load-balancing techniques are used, all bits of the Sequence Number MNA field can be varied.

Ext-Flags Field Format

Where fields are defined as follows:

• F - one-bit flag. When the flag is set to 1, it indicates the presence of the Flow ID field in the IOAM-DEX-ISD-MNA header.
• N - one-bit flag. When the flag is set to 1, it indicates the presence of the Sequence Number field in the IOAM-DEX-ISD-MNA header.
• U - unassigned one-bit flag. It MUST be zeroed on transmission and the value MUST be ignored upon receipt.

Considerations for IOAM and IOAM-DEX in MPLS Networks

Ingress-To-Egress Scope IOAM and IOAM-DEX Network Actions The I2E IOAM data fields carry the IOAM Option-Type(s) that require processing on the encapsulating and decapsulating nodes only. The IOAM Option-Type carried can be IOAM Edge-To-Edge Option-Type (value 3) defined in as well as IOAM-DEX Option-Type (value 4) defined in . The I2E IOAM data fields SHOULD NOT carry any IOAM Option-Type that require IOAM processing on the intermediate nodes as it will not be processed by them when IHS scope is set to "I2E, value 0x0". The I2E IOAM and IOAM-DEX Network Action procedure is summarized as follows:

• The encapsulating node inserts an MNA Sub-Stack with the MNA Label with IHS scope set to "I2E, value 0x0", one or more In-Stack Network Actions and one or more IOAM data fields in the MPLS packet.
• The intermediate nodes do not process the HBH IOAM data fields.
• The decapsulating node MAY punt the IOAM data fields from the packet with the receive timestamp to the slow path for processing. The receive timestamp is required by the various I2E OAM use-cases, including streaming telemetry. Note that the packet is not necessarily punted to the control-plane.
• The decapsulating node processes the IOAM data fields using the procedures defined in and . An example of IOAM processing is to export the IOAM data fields for streaming telemetry.
• The decapsulating node MUST remove the Network Actions and IOAM data fields from the received packet. The decapsulated packet is forwarded downstream or terminated locally similar to the regular data packets.

Hop-By-Hop Scope IOAM and IOAM-DEX Network Actions The HBH IOAM data fields carry the Option-Type(s) that require processing at the intermediate and/or encapsulating and decapsulating nodes. The IOAM Option-Type carried can be IOAM Pre-allocated Trace Option-Type (value 0), IOAM Incremental Trace Option-Type (value 1) and IOAM Proof of Transit (POT) Option-Type (value 2), and Edge-To-Edge Option-Type (value 3) defined in as well as IOAM-DEX Option-Type (value 4) defined in . Editor's note: IPv6 option is not supported for HBH IOAM Incremental Trace Option-Type (value 1). Similarly, MPLS network action is also not supported for HBH IOAM Incremental Trace Option-Type (value 1) and will be removed. The Hop-By-Hop IOAM and IOAM-DEX Network Action procedure is summarized as follows:

• The encapsulating node inserts an MNA Sub-Stack containing MNA Label, with IHS scope set to "HBH, value 0x1", one or more In-Stack Network Actions for IOAM, and one or more IOAM data fields in the MPLS packet.
• The intermediate node enabled with HBH IOAM function processes the data packet including the IOAM data fields as defined in and when the node recognizes the HBH scope in the MNA Sub-Stack.
• The intermediate node MAY punt the IOAM data fields from the packet with the receive timestamp to the slow path for processing when the node recognizes the HBH scope. The receive timestamp is required by the various HBH OAM use-cases, including streaming telemetry. Note that the packet is not necessarily punted to the control-plane.
• The intermediate node forwards the data packet downstream.
• The processing on the decapsulating node is the same as the I2E case.

Both HBH and I2E Scope IOAM may be carried in the Post-Stack MNA in an MPLS packet. In this case, the PSMH with HBH IOAM data fields MUST be added after the BOS and before the PSMH with I2E IOAM data fields. This way, the RLD required to process them on the intermediate nodes is minimized.

Node Capability The decapsulating node that needs to remove the IOAM and IOAM-DEX data fields and perform the IOAM and IOAM-DEX functions may not be capable of supporting it. The encapsulating node needs to know if the decapsulating node can support the IOAM and IOAM-DEX functions. The signaling extension for this capability exchange is outside the scope of this document. The intermediate node that is not capable of supporting the IOAM and IOAM-DEX functions defined in this document, can simply skip the IOAM and IOAM-DEX processing.

Nested MPLS Encapsulation When a packet is received with MPLS Encapsulated Network Action for IOAM and IOAM-DEX, the nested MPLS encapsulating node that needs to add different Network Action for IOAM and IOAM-DEX, the node MUST add a new MNA Sub-Stack with the Network Action for IOAM and IOAM-DEX as part of the new MPLS encapsulation.

Readable Label Depth Consideration The encapsulating node needs to make sure that the IOAM and IOAM-DEX data fields in MNA are added within the Readable Label Depth (RLD) of the downstream MNA capable nodes in order for them to be able to process the IOAM and IOAM-DEX.

Security Considerations Security considerations discussed in , , , and apply to this document. The usage of MPLS network actions defined in this document for IOAM and IOAM-DEX is intended for deployment in a single network administrative domain. As such, it assumes that the operator enabling the IOAM and IOAM-DEX operations has previously verified the integrity of the path. Still, operators need to properly secure the IOAM and IOAM-DEX in the domain to avoid malicious configuration and use, which could include injecting malicious IOAM and IOAM-DEX packets into the domain.

IANA Considerations

MPLS Network Action Opcodes IANA is requested to assign codepoints from its Network Action Opcodes registry (creation requested in and update requested in ) as specified in . MPLS Network Action Opcodes

Opcode	Description	In-Stack Only, Post-Stack Only, In-Stack and Post-Stack	Reference
TBA1	Network Action for IOAM and IOAM-DEX in PSD	In-Stack and Post-Stack	This document
TBA2	Network Action for IOAM-DEX in ISD	In-Stack Only	This document

Appendix A: Examples

In-Stack and Post-Stack Network Action Processing Order The In-Stack Network Action with Post-Stack Data can be added to interleave Post-Stack network actions with In-Stack network actions. The following example shows how to process the Post-Stack NA before some of the In-Stack NAs.

Post-Stack and In-Stack NA Processing Order

In this example, opcode 8 is processed first, followed by the opcode 1 for Flag-Based NAIs, then the Opcode TBA1 (which has corresponding Network Action opcode TBA1 in Post-Stack at an offset of 1 word, i.e., 4 bytes from the BOS), and finally opcode 7.

References Normative References Informational References IANA

Acknowledgments The authors would like to thank Adrian Farrel for reviewing this document and providing review comments. The authors would also like to thank Patrick Khordoc, Sagar Soni, Shwetha Bhandari, Vengada Prasad Govindan, Tarek Saad, Stewart Bryant, Xiao Min, Jaganbabu Rajamanickam, and Cheng Li for reviewing the early version of this document. The authors would also like to thank Mach Chen, Andrew Malis, Matthew Bocci, and Nick Delregno for the MPLS-RT expert review of the early version of this document.

Contributors The following people have substantially contributed to this document: