NETCONF Extension to support Trace Context propagation

Introduction Network automation and management systems commonly consist of multiple sub-systems and together with the network devices they manage, they effectively form a distributed system. Distributed tracing is a methodology implemented by tracing tools to follow, analyze and debug operations, such as configuration transactions, across multiple distributed systems. An operation is uniquely identified by a trace-id and through a trace context, carries some metadata about the operation. Propagating this "trace context" between systems enables forming a coherent view of the entire operation as carried out by all involved systems. The W3C has defined two HTTP headers for context propagation that are useful in use case scenarios of distributed systems like the ones defined in . This document defines an extension to the NETCONF protocol to add the same concepts and enable trace context propagation over NETCONF. It is worth noting that the trace context is not meant to have any relationship with the data that is carried with a given operation (including configurations, service identifiers or state information). A trace context also differs from in several ways as the trace operation may involve any operation (including for example validate, lock, unlock, etc.) Additionally, a trace context scope may include the full application stack (orchestrator, controller, devices, etc) rather than a single NETCONF server, which is the scope for the transaction-id. The trace context is also complemetary to as a given trace-id can can associated to the different transaction-ids as part of the information exported to the collector. The following enhancement of the reference SDN Architecture from RFC 8309 shows the impact of distributed traces for a network operator.

| | .------------------. | | . : . | | . : . | Collector | ------------ ------------ ------------ | (Metrics, | | | | | | | | Events, | | Controller | | Controller | | Controller | --> | Logs, | | | | | | | | Traces) | ------------ ------------ ------------ | | : . . : | | : . . : | | : . . : | | --------- --------- --------- --------- | | | Network | | Network | | Network | | Network | | | | Element | | Element | | Element | | Element | -> | | --------- --------- --------- --------- ------------- Figure 1: A Sample SDN Architecture from RFC8309 augmented to include the export of metrics, events, logs and traces from the different components to a common collector. ]]> The network automation, management and control architectures are distributed in nature. In order to "manage the managers", operators would like to use the same techniques as any other distributed systems in their IT environment. Solutions for analysing Metrics, Events, Logs and Traces (M.E.L.T) are key for the successful monitoring and troubleshooting of such applications. Initiatives such as the OpenTelemetry enable rich ecosystems of tools that NETCONF-based applications would want to participate in. With the implementation of this trace context propagation extension to NETCONF, backend systems behind the M.E.L.T collector will be able to correlate information from different systems but related to a common context.

Implementation example 1: OpenTelemetry We will describe an example to show the value of trace context propagation in the NETCONF protocol. In Figure 2, we show a deployment based on Figure 1 with a single controller and two network elements. In this example, the NETCONF protocol is running between the Orchestrator and the Controller. NETCONF is also used between the Controller and the Network Elements. Let's assume an edit-config operation between the orchestrator and the controller that results (either synchronously or asynchronously) in corresponding edit-config operations from the Controller towards the two network elements. All trace operations are related and will create M.E.L.T data.

| | .------------------. | | . NETCONF | | . edit-config (trace-id "1", parent-id "A") | Collector | ------------ | (Metrics, | | | | Events, | | Controller | ------------------------------------> | Logs, | | | OTLP protocol | Traces) | ------------ | | : . NETCONF | | : . edit-config (trace-id "1", parent-id "B") | | : . | | --------- --------- | | | Network | | Network | OTLP protocol | | | Element | | Element | --------------------------> | | --------- --------- ------------- Figure 2: An implementation example where the NETCONF protocol is used between the Orchestrator and the Controller and also between the Controller and the Network Elements. Every component exports M.E.L.T information to the collector using the OTLP protocol. ]]> Each of the components in this example (Orchestrator, Controller and Network Elements) is exporting M.E.L.T information to the collector using the OpenTelemetry Protocol (OTLP). For every edit-config operation, the trace context is included. In particular, the same trace-id "1" (simplified encoding for documentation) is included in all related NETCONF messages, which enables the collector and any backend application to correlate all M.E.L.T messages related to this transaction in this distributed stack. Another interesting attribute is the parent-id. We can see in this example that the parent-id between the orchestrator and the controller ("A") is different from the one between the controller and the network elements ("B"). This attribute will help the collector and the backend applications to build a connectivity graph to understand how M.E.L.T information exported from one component relates to the information exported from a different component. With this additional metadata exchanged between the components and exposed to the M.E.L.T collector, there are important improvements to the monitor and troubleshooting operations for the full application stack.

Implementation example 2: YANG DataStore OpenTelemetry implements the "push" model for data streaming where information is sent to the back-end as soon as produced and is not required to be stored in the system. In certain cases, a "pull" model may be envisioned, for example for performing forensic analysis while not all OTLP traces are available in the back-end systems. An implementation of a "pull" mechanism for M.E.L.T. information in general and for traces in particular, could consist of storying traces in a yang datastore (particularly the operational datastore.) Implementations should consider the use of circular buffers to avoid resources exhaustion. External systems could access traces (and particularly past traces) via NETCONF, RESTCONF, gNMI or other polling mechanisms. Finally, storying traces in a YANG datastore enables the use of YANG-Push or gNMI Telemetry as an additional "push" mechanisms. This document does not specify the YANG module in which traces could be stored inside the different components. That said, storing the context information described in this document as part of the recorded traces would allow back-end systems to correlate the information from different components as in the OpenTelemetry implementation. Note to be removed in the future: Some initial ideas are under discussion in the IETF for defining a standard YANG data model for traces. For example see: I-D.quilbeuf-opsawg-configuration-tracing which focusses only on configuration change root cause analysis use case (see bellow the use case desciption.). This ideas are complementary to this draft.

| | .------------------. pull or push | | . NETCONF | | . edit-config (trace-id "1", parent-id "A") | Collector | ---------------- | (Metrics, | | | NC/RC/gNMI or YP | Events, | | Controller | --------------------------------> | Logs, | | YANG DataStore| pull or push | Traces) | ---------------- | | : . NETCONF | | : . edit-config (trace-id "1", parent-id "B") | | : . | | --------- --------- | | | Network | | Network | NC/RC/gNMI or YP | | | Element | | Element | --------------------------> | | | YG DS | | YG DS | pull or push | | --------- --------- ------------- Figure 3: An implementation example where the NETCONF protocol is used between the Orchestrator and the Controller and also between the Controller and the Network Elements. M.E.L.T. information is stored in local Yang Datastores and accessed by the collector using "pull" mechanisms using the NETCONF (NC), RESTCONF (RC) or gNMI protocols. A "push" strategy is also possible via YANG-Push or gNMI. ]]>

Use Cases

Provisioning root cause analysis When a provisioning activity fails, errors are typically propagated northbound, however this information may be difficult to troubleshoot and typically, operators are required to navigate logs across all the different components. With the support for trace context propagation as described in this document for NETCONF, the collector will be able to search every trace, event, metric, or log in connection to that trace-id and faciliate the performance of a root cause analysis due to a network changes. The trace information could also include as an optional resource the different NETCONF transaction ids described in .

System performaednce profiling When operating a distributed system such as the one shown in Figure 2, operators are expected to benchmark what are the Key Performance Indicators (KPIs) for the most common tasks. For example, what is the typical delay when provisioning a VPN service across different controllers and devices. Thanks to Application Performance Management (APM) systems, from these KPIs, an operator can detect a normal and abnormal behaviour of the distributed system. Also, an operator can better plan any upgrades or enhancements in the platform. With the support for context propagation as described in this document for NETCONF, much richer system-wide KPIs can be defined and used for troubleshooting as the metrics and traces propagated by the different components share a common context. Troubleshooting for abnormal behaviours can also be troubleshot from the system view down to the individual element.

Billing and auditing In certain circuntances, we could perceive that tracing infomration could be used as additional inputs to billing systems. In particular, trace context information could be used to validate that a certain northbound order was carried out in southbound systems.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT","SHOULD","SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. The XML prefixes used in this document are mapped as follows: xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0", xmlns:notif="urn:ietf:params:xml:ns:netconf:notification:1.0", xmlns:yp="urn:ietf:params:xml:ns:yang:ietf-yang-patch" and xmlns:ypatch="urn:ietf:params:xml:ns:yang:ietf-yang-patch".