]> A Data Manifest for Contextualized Telemetry Data Huawei
benoit.claise@huawei.com
Huawei
jean.quilbeuf@huawei.com
Telefonica I+D
Don Ramon de la Cruz, 82 Madrid 28006 Spain diego.r.lopez@telefonica.com
Telefonica I+D
Ronda de la Comunicacion, S/N Madrid 28050 Spain ignacio.dominguezmartinez@telefonica.com
Swisscom
Binzring 17 Zurich 8045 Switzerland thomas.graf@swisscom.com
OPS OPSAWG Network platforms use Model-driven Telemetry, such as YANG-Push, to continuously stream information, including both counters and state information. This document describes the metadata that ensure that the collected data can be interpreted correctly. This document specifies the data manifest, composed of two YANG data models (the platform manifest and the data collection manifest). These YANG modules are specified at the network (e.g. controller) level to provide a model that encompasses several network platforms. The data manifest must be streamed and stored along with the data, up to the collection and analytics systems in order to keep the collected data fully exploitable by the data scientists and relevant tools. Additionally, this document proposes an augmentation of the YANG-Push model to include the actual collection period, in case it differs from the configured collection period.
Introduction Network platforms use Model-driven Telemetry (MDT), such as YANG-Push , to continuously stream information, including both counters and state information. This document specifies what needs to be kept as metadata to ensure that the collected data can still be interpreted correctly throughout the collection and network analytics toolchain. When streaming YANG-structured data with YANG-Push , there is a semantic definition in the corresponding YANG module definition. This is the semantic information for the collected data nodes: While this semantic is absolutely required to correctly decode and interpret the data, understanding the network platform and collection environment contexts information is equally important to interpret the data. One part of this information is the actual collection period, as opposed to the configured collection period. On some platforms, that period can be adjusted automatically by the platform, for instance to reduce the load incurred by sending the telemetry. To later exploit the collected data, getting this actual collection period is crucial. This document defines a YANG model augmenting the YANG-Push model to expose the actual collection period in . This document introduces the data manifest, which is composed of two YANG data models, namely, the platform manifest and the data collection manifest, in order to keep the collected data exploitable by the data scientists and relevant tools. The platform manifest contains information characterizing the platform streaming the telemetry information, while the data collection manifest contains the required information to characterize how and when the telemetry information was metered. The platform manifest is specified in . The data collection manifest is specified in . The two proposed YANG modules for the data manifest do not expose any new information but rather define what should be exposed by a platform streaming or storing telemetry data. Some related YANG modules have been specified to retrieve the platform capabilities such as:
  • "YANG Library" .
  • "YANG Modules Describing Capabilities for Systems and Datastore Update Notifications" for the platform capabilities regarding the production and export of telemetry data.
  • , which is based on to define the optimal settings to stream specific items (i.e., per path).
These related YANG modules are important to discover the capabilities before applying the telemetry configuration (such as on-change subscription). Some of their content is part of the context for the streamed data. This document covers only metadata about the collection context for the telemetry. The collected data is likely to be transformed into usable indicators for the network. The list of such transformation operations applied to the data is often called data lineage. Supplying the data lineage for the computed indicators is out of scope of this document. To retrieve the context in which a particular piece of data was collected, three elements are necessary: the time of data emission, the originating platform and the subscription through which the data arrived. The approach proposed in this document delegates the time retrieval to the database storing the collected telemetry and focusing on providing a way to match a platform and a subscription identifier to the collection context. This is consistent with most of the YANG modules for devices, which focus on describing the current state of the device, rather than the evolution of that state through time.
Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. Platform: equipment of the network able to produce telemetry. Data manifest: The necessary data required to interpret a telemetry information. Platform manifest: part of the data manifest that completely characterizes the platform producing the telemetry information Data collection manifest: part of the data manifest that completely characterizes how and when the telemetry information was metered. Datapoint: an instance of data collected via telemetry at a specific time. Collector: software that receives the stream of telemetry.
Use Cases
Network Analytics Streamed information from network platforms is used for network analytics, incident detection, and in the closed control loop for network automation. See for definition of some of these terms. This streamed data can be stored in a database (sometimes called a big data lake) for further analysis. As an example, a database could store a time series representing the evolution of a specific counter collected from a network platform. When analyzing the data, a network operator/data scientist must understand the context information for these data:
  • This counter definition, typically as defined in the YANG model.
  • The network platform vendor, model, and OS.
  • The collection parameters.
Characterizing the source used for producing the data (vendor, platform, and OS) is useful to complement the data. As an example, knowing the exact data source software specification might reveal a particularity in the observed data, explained by a specific bug, a specific bug fix, or simply a particular specific behavior. This is also necessary to ensure the reliability of the collected data. On top of that, in particular for YANG-Push , it is crucial to know the set of YANG modules supported by the platform, along with their deviations. In some cases, there might even be some backwards incompatible changes in native modules (i.e., vendor proprietary modules) between one OS version to the next one. This information is captured by means of the platform manifest . From a collection parameters point of view, the data scientists analyzing the collected data must know whether the counter was requested from the network platform as on-change or at specific cadence . Indeed, an on-change collection explains why there is a single value as opposed to a time series. In case of periodic collection, this exact cadence might not be observable in the time series. Indeed, this time series might report some values as 0 or might even omit some values. The reason for this behavior might be diverse: the network platform was under stress, with a too small observation period, compared to the minimum-observed-period . Knowing the conditions under which the counter was collected and streamed (along with the platform details) help drawing the right conclusions. As an example, taking into account the value of 0 might lead to a wrong conclusion that the counter dropped to zero.
New Device Onboarding When a new device is onboarded, operators have to check that the new device streams data with YANG-Push, that the telemetry data is the right one, that the data is correctly ingested in the collection system, and finally that the data can be analyzed (compared with other similar devices). For the last point, the data manifest, which must be linked to the data up to the collection and analytics system, contains the relevant information.
Data Mesh Principles in Networking The concept behind the data mesh are:
  • Domain Ownership: Architecturally and organizationally align business, technology, and analytical data, following the line of responsibility. The Data Mesh principles adopt the boundary of bounded context to individual data products where each domain is responsible for (and owns) its data and models.
  • Data as a Product: The “Domain” owners are responsible to provide the data in useful way (discoverable through a catalog, addressable with a permanent and unique address, understandable with well-defined semantics, trustworthy and truthful, self-describing for easy consumption, interoperable by supporting standards, secure, self-contained, etc.) and should treat consumers of that data as customers. It requires and relies on the “Domain Ownership” principle.
  • Self-serve Data Platform: This fosters the sharing of cross-domain data in order to create extra value.
  • Federated Computational Governance: Describes the operating model and approach to establishing global policies across a mesh of data products.
The most relevant concept for this document is the "Data as a Product" principle. The data manifest fulfills this principle as the two YANG data models, platform manifest and the data collection manifest, along with the data, provide all the necessary information in a self-describing way for easy consumption.
The "ietf-yp-current-period" YANG module As explained earlier, the collection period is crucial information for a posteriori interpretation of the collected telemetry. Some platforms will adjust the collection period depending on their capabilities and current load. The YANG module proposed in this section augments the "ietf-subscribed-notification" module to provide the "current-period" leaf. The value of this leaf indicates the current collection which might be different from the configured collection period. contains the YANG tree diagram of the "ietf-yp-current-period" module.
YANG tree diagram for "ietf-yp-current-period" module
The code of the "ietf-yp-current" YANG module is given below. WG List: Author: Benoit Claise Author: Jean Quilbeuf Author: Diego R. Lopez Author: Ignacio Dominguez Author: Thomas Graf "; description "This module augments ietf-subscribed-notification and ietf-yang-push with the current-period statistics reporting the actual collection period, as opposed to the configured one. Copyright (c) 2025 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Revised BSD License set forth in Section 4.c of the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices. "; revision 2025-02-21 { description "Initial revision"; reference "RFC xxxx: A Data Manifest for Contextualized Telemetry Data"; } augment "/sn:subscriptions/sn:subscription" { description "Adds current period statistics"; leaf current-period { when '../yp:periodic'; type yp:centiseconds; description "Period during two successive data collections, in the current state. Might differ from the configured period when the platform might increase the period automatically when it is overloaded."; } } } ]]>
Platform Manifest
Overview of the Model contains the YANG tree diagram of the ietf-platform-manifest module.
YANG tree diagram for ietf-platform-manifest module ../../module/name | +--ro import-only-module* [name revision] | +--ro name yang:yang-identifier | +--ro revision union | +--ro namespace inet:uri | +--ro location* inet:uri | +--ro submodule* [name] | +--ro name yang:yang-identifier | +--ro revision? revision-identifier | +--ro location* inet:uri +--ro schema* [name] | +--ro name string | +--ro module-set* -> ../../module-set/name +--ro datastore* [name] +--ro name ds:datastore-ref +--ro schema -> ../../schema/name ]]>
The YANG module contains a list of platform manifests (in 'platforms/platform'), indexed by the identifier of the platform. That identifier should be defined by the network manager so that each platform emitting telemetry has a unique identifier. There are several ongoing documents about managing the inventory of the network , based on . The platform identifier should be the same as the identifier used in inventories or the 'node-id' in . As an example, the identifier could be the 'sysName' from the ietf-notification module presented in . The scope of the "ietf-platform-manifest" module is the scope of the data collection, i.e., a given network, therefore it contains a collection of platform manifests, as opposed to the device scope, which would contain a single platform manifest. The platform manifest is characterized by a set of parameters ('name', 'software-version', 'software-flavor', 'os-version', 'os-type') that are aligned with the YANG Catalog so that the YANG Catalog could be used to retrieve the YANG modules a posteriori. The vendor of the platform can be identified via its name 'vendor' or its PEN number 'vendor-pen', as described in . The platform manifest also includes the contents of the YANG Library . That module set is particularly useful to retrieve the YANG modules associated to a subscription by analyzing the xpath filters or the subtree filters. Xpath filters are based on module names (see , description of leaf 'stream-xpath-filter', page 45). Subtree filters are based on namespaces. The platform manifest is obtained by specifying the new fields defined above and mounting the YANG library module, along with the YANG Revisions augmentations. Thus, the YANG Library part is not repeated in the YANG module for the platform manifest.
YANG module ietf-platform-manifest This section defines the ietf-platform manifest YANG module. WG List: Author: Benoit Claise Author: Jean Quilbeuf Author: Diego R. Lopez Author: Ignacio Dominguez Author: Thomas Graf "; description "This module describes the platform information to be used as context of data collection from a given network element. The contents of this model must be streamed along with the data streamed from the network element so that the platform context of the data collection can be retrieved later. The data content of this model should not change except on upgrade or patching of the device. Copyright (c) 2022 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Revised BSD License set forth in Section 4.c of the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices. "; revision 2025-02-21 { description "Initial revision"; reference "RFC xxxx: A Data Manifest for Contextualized Telemetry Data"; } grouping platform-details { description "This grouping contains the information about a particular platform, as stored in the YANG catalog."; leaf name { type string { length "1..1023"; } description "Model of the platform from which data is collected."; } leaf vendor { type string { length "1..1023"; } description "Organization that implements that platform."; } leaf vendor-pen { type uint32; description "Vendor's registered Private Enterprise Number"; reference "RFC9371: Registration Procedures for Private Enterprise Numbers (PENs)"; } leaf software-version { type string { length "1..1023"; } description "Name of the version of software. With respect to most network device appliances, this will be the operating system version. But for other YANG module implementation, this would be a version of appliance software. Ultimately, this should correspond to a version string that will be recognizable by the consumers of the platform."; } leaf software-flavor { type string { length "1..1023"; } description "A variation of a specific version where YANG model support may be different. Depending on the vendor, this could be a license, additional software component, or a feature set."; } leaf os-version { type string { length "1..1023"; } description "Version of the operating system using this module. This is primarily useful if the software implementing the module is an application that requires a specific operating system version."; } leaf os-type { type string { length "1..1023"; } description "Type of the operating system using this module. This is primarily useful if the software implementing the module is an application that requires a specific operating system type."; } } container platforms { config false; description "Top container including all platforms in scope. If this model is hosted on a single device, it should contain a single entry in the list. At the network level, it should contain an entry for every monitored platform."; list platform { key "id"; description "Contains information about the platform that allows identifying and understanding the individual data collection information."; leaf id { type string { length "1..1023"; } description "Identifies a given platform on the network, for instance the 'sysName' of the platform. The 'id' has to be unique within the network scope at every point in time. The same id can point to different platform if they are not simultaneously part of the network, e.g., when a device associated to a particular id is replaced."; } uses platform-details; uses yanglib:yang-library-parameters; } } } ]]>
Data Collection Manifest
Overview of the Model contains the YANG tree diagram of the "example-collection-manifest" module. The module relies upon the YANG Schema mount to reuse existing YANG modules describing the current data collection status. This module is an example as YANG Schema mount does not support design-time schema mount. explains how the YANG tree is obtained.
YANG tree diagram for example-collection-manifest module /p-mf:platforms/p-mf:platform/p-mf:id +--ro streams/ | +--ro stream* [name] | +--ro name string | +--ro description? string +--ro filters/ | +--ro stream-filter* [name] | | +--ro name string | | +--ro (filter-spec)? | | +--:(stream-subtree-filter) | | +--:(stream-xpath-filter) | | +--ro stream-xpath-filter? yang:xpath1.0 | | {xpath}? | +--ro selection-filter* [filter-id] | +--ro filter-id string | +--ro (filter-spec)? | +--:(datastore-subtree-filter) | +--:(datastore-xpath-filter) | +--ro datastore-xpath-filter? yang:xpath1.0 | {sn:xpath}? +--ro subscriptions/ +--ro subscription* [id] +--ro id subscription-id +--ro (target) | +--:(stream) | | +--ro (stream-filter)? | | | +--:(by-reference) | | | | +--ro stream-filter-name | | | | stream-filter-ref | | | +--:(within-subscription) | | | +--ro (filter-spec)? | | | +--:(stream-subtree-filter) | | | +--:(stream-xpath-filter) | | | +--ro stream-xpath-filter? | | | yang:xpath1.0 {xpath}? | | +--ro stream stream-ref | +--:(datastore) | +--ro datastore identityref | +--ro (selection-filter)? | +--:(by-reference) | | +--ro selection-filter-ref | | selection-filter-ref | +--:(within-subscription) | +--ro (filter-spec)? | +--:(datastore-subtree-filter) | +--:(datastore-xpath-filter) | +--ro datastore-xpath-filter? | yang:xpath1.0 {sn:xpath}? +--ro stop-time? yang:date-and-time +--ro encoding? encoding +--ro receivers | +--ro receiver* [name] | +--ro name string | +--ro sent-event-records? | | yang:zero-based-counter64 | +--ro excluded-event-records? | | yang:zero-based-counter64 | +--ro state enumeration +--ro (update-trigger)? | +--:(periodic) | | +--ro periodic! | | +--ro period centiseconds | | +--ro anchor-time? yang:date-and-time | +--:(on-change) {on-change}? | +--ro on-change! | +--ro dampening-period? centiseconds | +--ro sync-on-start? boolean | +--ro excluded-change* change-type +--ro current-period? yp:centiseconds ]]>
The 'data-collections' container contains the information related to each YANG-Push subscription. As for the platform manifest, these subscriptions are indexed by the 'platform-id', so that all subscriptions in the network can be represented at the network level without any conflict. As most of the information related to YANG-push subscription and is stored in the "ietf-yang-push" module, these modules are mounted. These modules have a part common to all subscriptions of the platform, stored in the 'streams' and 'filters' containers. The information about subscriptions themselves are stored in the 'subscriptions/subscription' list, indexed by a subscription identifier. In the subscription object, the 'current-period' indicates the period currently used between two updates. That leaf can only be present when the subscription is periodic. The current period might differ from the requested period if the platform implements a mechanism to increase the collection period when it is overloaded. Having the current period information is crucial to understand if telemetry is missing because of a bug or a packet loss or simply because it was dynamically adjusted by the platform. The 'current-period' data node is added by the module 'ietf-data-collection-manifest-statistics' presented in . This module augments the subscription list from the module 'ietf-subscribed-notifications'. It is mounted as well via the YANG Schema Mount mechanism. The module for the data collection manifest is presented in .
The "example-collection-manifest" YANG module This section includes the code of the "example-collection-manifest" YANG module. Additionally, it defines the extension data file for YANG schema mount. The data collection manifest MUST conform to the model obtained by combining these two specifications. WG List: Author: Benoit Claise Author: Jean Quilbeuf Author: Diego R. Lopez Author: Ignacio Dominguez Author: Thomas Graf "; description "This module describes the context of data collection from a given network element. The contents of this model must be streamed along with the data streamed from the network element so that the context of the data collection can be retrieved later. This module must be completed with ietf-platform-manifest to capture the whole context of a data collection session. The key words 'MUST', 'MUST NOT', 'REQUIRED', 'SHALL', 'SHALL NOT', 'SHOULD', 'SHOULD NOT', 'RECOMMENDED', 'NOT RECOMMENDED', 'MAY', and 'OPTIONAL' in this document are to be interpreted as described in BCP 14 (RFC 2119) (RFC 8174) when, and only when, they appear in all capitals, as shown here. Copyright (c) 2025 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Revised BSD License set forth in Section 4.c of the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices. "; revision 2025-02-21 { description "Initial revision"; reference "RFC XXXX: A Data Manifest for Contextualized Telemetry Data"; } container data-collections { config false; description "Contains the configuration and statistics for the collected data, per node in the network."; list data-collection { key "platform-id"; description "Defines the information for each collected object"; leaf platform-id { type leafref { path "/p-mf:platforms/p-mf:platform/p-mf:id"; } description "Identifier of the platform collecting the data. This identifier is the same as the one in the platform manifest."; } yangmnt:mount-point "yang-push-collection" { description "This mount point MUST mount the following modules and their dependencies: * ietf-subscribed-notifications * ietf-yang-push * ietf-yp-current-period. This mount point MUST NOT mount any other modules."; reference "RFC8639: Subscription to YANG Notifications RFC8641: Subscription to YANG Notifications for datastore updates"; } } } } ]]>
Data Manifest and the Collected Data This section focuses on relating the collected data to the data manifest. As this document specifically focuses on giving context on data collected via streamed telemetry, it is assumed that a streaming telemetry system is available. Another premise of this document is the storage of the collected data into a database for later exploitation. It is assumed that such a database exists and can be used for storing the data manifest.
Collecting the Data Manifest The data manifest MUST be streamed and stored along with the collected data. In case the collected data are moved to a different place (typically a database), the companion data manifest MUST follow the collected data. Storing the collected data without the companion data manifest might prevent the correct interpretation of the collected data. The data manifest MUST be updated when the data manifest information changes, for example, when a router is upgraded, when a new telemetry subscription is configured, or when the telemetry subscription parameters change. The data manifest can itself be considered as a time series, and stored in a similar fashion to the collected data. This document recommends reusing the existing telemetry system (in-band approach) in order to lower the efforts for implementing this approach. To enable a platform supporting streaming telemetry to also support the data manifest, it is sufficient that this platform supports the models from Sections and . The collection of the data manifest MUST be explicitly configured by the collector by requesting the relevant subscriptions. These subscriptions MUST include the platform manifest and the data collection manifest, possibly limited to the subscriptions for which the context needs to be retrieved a posteriori. shows how the in-band approach would work while storing to a time-series database (TSDB). Each type of manifest has its own rough frequency update, i.e. at reboot for the platform manifest and when subscriptions are modified for the data collection manifest. The data manifest SHOULD be streamed with the YANG-Push on-change feature (also called event-driven telemetry) whenever possible. A platform manifest is likely to remain the same until the platform is updated. Thus, the platform manifest only needs to be collected once per streaming session and updated after a platform reboot. The "subscription-terminated" will indicate to the collector that the platform rebooted. The collector MUST then collect the potential update of the platform manifest on re-establishment of the subscription. Using the on-change feature enables to capture dynamic changes to the platform manifest as well, if any. Regarding the data manifest, the elements common to all subscriptions, such as the stream definitions and the common filters might be updated less frequently than the subscriptions. Relying on YANG-Push on-change feature enables keeping an up-to-date version of the data collection manifest. The underlying time series database should accommodate the various rates at which different parts of the data manifest are updated. In particular, storing the platform manifest should be optimized to avoid duplicating repeated content and only storing a new version when there is a change in the manifest.
Mapping Collected Data to the Data Manifest As explained in the introduction, three elements are necessary to identify the data manifest associated to a datapoint:
  • the time at which the data was sent from the device,
  • the originating platform sending the data, and
  • the identifier of the subscription that produced the data.
This elements can be either known to the collector, if it is the one configuring the collection, or retrieved via dedicated headers as proposed in . In order to enable a posteriori retrieval of the data manifest associated to a datapoint, the collector MUST keep the subscription identifier and platform identifier in the metadata of the collected values. With this information, to retrieve the data manifest from a datapoint, the following happens:
  • The subscription identifier, platform identifier and time stamp of the data are retrieved from the datapoint metadata
  • The platform manifest for that datapoint is obtained by looking up the latest version before the time stamp matching the platform identifier.
  • The data collection manifest for that datapoint is obtained by looking up the latest version before the time matching the platform identifier and the subscription identifier.
The reliability of the collection of the data manifest is the same as the reliability of the data collection itself, since the data manifest is like any other data.
Operational Considerations It is expected that the data manifest is streamed directly from the network equipment, along with YANG-Push data. However, if the network equipment streaming telemetry does not yet support the YANG modules from the data manifest specified in this document, the telemetry collector could populate the data manifest from available information collected from the platform. This latter option requires efforts on the telemetry collector side, as the information gathered in the data manifest proposed in this document could be scattered among various standard and vendor-specific YANG modules , that depend on the platform.
Example shows an example of both a Platform manifest and corresponding data collection manifests. The list of YANG modules in the yang-library container is kept empty for brevity.
Example of data manifest
contains the data collection manifest for two XPaths subscriptions. With the data collection manifest for the first one, with subscription identifier 4242, the exact semantics of the collected path, here the administrative status of the network interfaces, can be obtained by looking up the module in the yang-library of the corresponding platform manifest, in order to obtain the exact revision of ietf-interfaces used at collection time. Also, the "on-change" container indicates that data will be sent only if there is a change, thus not receiving data indicates that the administrative status of the interface did not change. The other example of data collection manifest, with subscription identifier 4243, shows how a periodic subscription is reported. In that example, the 'current-period' indicates that the requested period of 10s (1000 centiseconds) could not be attained and is now of 20s, for instance because the device is overloaded. gives the command line for validating this example using .
Security Considerations The YANG modules specified in this document define a schema for data that is designed to be accessed via network management protocols such as NETCONF or RESTCONF . The lowest NETCONF layer is the secure transport layer, and the mandatory-to-implement secure transport is Secure Shell (SSH) . The lowest RESTCONF layer is HTTPS, and the mandatory-to-implement secure transport is TLS . The Network Configuration Access Control Model (NACM) provides the means to restrict access for particular NETCONF or RESTCONF users to a preconfigured subset of all available NETCONF or RESTCONF protocol operations and content. Some of the readable data nodes in this YANG module may be considered sensitive or vulnerable in some network environments. It is thus important to control read access (e.g., via get, get-config, or notification) to these data nodes. These are the subtrees and data nodes and their sensitivity/vulnerability:
  • ietf-plaftorm-manifest:platforms/platform contains details about the platform that an attacker could use to find the known vulnerabilities of the platform.
As the present approach reuses an existing telemetry system, the security considerations lie with the new content divulged in the new manifests. Appropriate access control filters must be associated to the corresponding leafs and containers, as well as the databases storing them. The integrity and provenance of the data of the collection manifest can be ensured by a signing mechanism such as .
IANA Considerations RFC Ed.: replace XXXX with actual RFC number and remove this note. IANA is requested to register the following URIs in the "ns" subregistry within the "IETF XML Registry" : URI: urn:ietf:params:xml:ns:yang:ietf-platform-manifest Registrant Contact: The IESG. XML: N/A; the requested URI is an XML namespace. URI: urn:ietf:params:xml:ns:yang:ietf-yp-current-period Registrant Contact: The IESG. XML: N/A; the requested URI is an XML namespace. IANA is requested to register the following YANG modules in the "YANG Module Names" subregistry within the "YANG Parameters" registry. Name: ietf-platform-manifest Maintained by IANA? N Namespace: urn:ietf:params:xml:ns:yang:ietf-platform-manifest Prefix: p-mf Reference: RFC XXXX Name: ietf-yp-current-period Maintained by IANA? N Namespace: urn:ietf:params:xml:ns:yang:ietf-yp-current-period Prefix: yp-cp Reference: RFC XXXX
Contributors
Open Issues
  • Do we want to handle the absence of values, i.e. add information about missed collection or errors in the collection context ? It could also explain why some values are missing. On the other hand, this might also be out scope. CLOSED: the goal of the manifest is to be able to detect miscollection a posteriori. Assurance of the metric collection is out of scope and could be done via an external mechanism such as SAIN.
  • Henk: how does this interact with SBOM effort? CLOSED: SBOM is another kind of manifest, we are focusing here on data collection.
  • What is the link with the RFC8345 NodeId and IVY? CLOSED: added text.
  • Handling of deletion in . CLOSED: out of scope
Datamesh Architecture Yanglint
Changes between revisions v05 -> v06
  • Example can be validated using yanglint
  • Applied details comments from Joe and Med
  • Making the "current-period" update more generic and mentioning it in the introduction
  • Section 7 (previously 5) reworked to clarify how data manifest is collected and retrieved from a datapoint
  • Remove use of YANG schema mount for the platform manifest and change data collection manifest to example
v04 -> v05
  • Remove references to full-include draft, use schema mount.
  • Explain link with schema node id
v03 -> v04
  • State that data lineage is out of scope
  • Replace copy-pasted version of the modules with schema mount version, use full-embed for the "real" one
  • Schema mount version is the fallback plan if full:embed is not there fast enough.
  • Update examples accordingly
v02 -> v03
  • Explicit that modules are network (Controller) level
  • InfluxDB example changed to TSDB example aligned with
  • Minor edits i.e. network element -> platform , object -> data node
v01 -> v02
  • Updated example with latest version of the model.
v00 (WG adoption) - v01
  • Solve integrity issue by delegating to .
v05 -> v06
  • Remove YANG packages
  • Switch YANG models from device view to network view
  • Add PEN number to identify vendors
  • Intro rewritten with uses cases
  • Added an "Operational Considerations" section
  • Switch from MDT to YANG-push
v04 -> v05
  • First version of example scenario
  • Updated affiliation
  • Updated YANG module names to ietf-platform-manifest and ietf-data-collection-manifest
  • Unify used terms as defined in the terminology section
  • Replaced 'device' with 'platform'
  • Split Section 5 into two sections for better readibility
v03 -> v04
  • Fix xym error
  • Moved terminology after introduction
  • Clarified the role of the module
v02 -> v03
  • Add when clause in YANG model
  • Fix validation errors on YANG modules
  • Augment YANG library to handle semantic versioning
v01 -> v02
  • Alignment with YANGCatalog YANG module: name, vendor
  • Clarify the use of YANG instance file
  • Editorial improvements
v00 -> v01
  • Adding more into data platform: yang packages, whole yanglib module to specify datastores
  • Setting the right type for periods: int64 -> uint64
  • Specify the origin datastore for mdt subscription
  • Set both models to config false
  • Applying text comments from Mohamed Boucadair
  • Adding an example of data-manifest file
  • Adding rationale for reusing telemetry system for collection of the manifests
  • Export manifest with on change telemetry as opposed to YANG instance file
v00
  • Initial version
An Example of Use Based on MDT In this example, the goal is to collect the administrative status and number of received bytes for the interfaces of a fictional ACME device, and store the result in a time-series database. The metrics are collected using YANG-Push, which is configured by specifying their XPaths and when they should be collected (periodically or on-change). More precisely, the Xpaths to collect are "ietf-interfaces:interfaces/interface/enabled" on every change and "ietf-interfaces:interfaces/interface/statistics/in-octets" every 100 milliseconds. The paths here are referring to the YANG module from . The configuration of YANG push is out of scope for this document. Since they don’t have the same trigger, each of the path must be collected in its own subscription. presents an example for such a collection.
Example of Collection From a Device to a TSDB | TSDB | | Collector | +--------+ +------------+ ^ | | +---------+ | Device | +---------+ ]]>
In the scenario depicted in , the collector receives YANG-push data from the device and stores it into a TSDB. This section first presents a version without data manifest and then how to enrich it with the data manifest. Examples rely on the notation from to represent how the data is stored in the TSDB. Without the data manifest, the result of the collection would be stored as showed in . The "host" label indicates the devices from which the data is collected and the YANG keys are included as well. Here the interface "eth0" is enabled and received 1234 octets. In that case, the value is stored, without any way to know how the value was obtained.
Storing Datapoints without Data Manifest
An option for keeping the data manifest with the data is to store it directly into the TSDB. In that case, the collector can subscribe to the data exported by the module presented in this document and store it as other metrics. For the platform manifest, assuming the platform identifier is "PE1", the collector subscribes to the path "ietf-platform-manifest:platforms/platform[id=PE1]". For the data collection manifests, the collector subscribes to the path "ietf-data-collection-manifest:data-collections/data-collection[platform-id="PE1"]/yang-push-collection/subscriptions/subscription[id=X]" where X is the subscription identifier of existing subscriptions. With the approach from , the corresponding subtrees would be split into a set of datapoints, one per leaf. shows two examples of storing leaves in a TSDB. The first leaf is the vendor PEN number, which is part of the platform manifest. The second leaf is the Xpath filter used for subscription to the interface status.
Example of storing Platform and Data Collection Manifest: Vendor PEN and Xpath filter.
In the labels, the "host" might be different from the "platforms_platform_id" in case the collector is the one assembling it, i.e. for devices that do not natively support the data manifest. In that case, the value of this label could be the hostname of the collector. The host value does not matter for retrieving the data manifest as the platform identifier is the meaningful field. In this example, retrieving the platform manifest associated to a collected datapoint is done by looking for datapoints that have the label "platforms_platform_id" equal to the value of the host for that collected datapoint. In order to link a datapoint with the corresponding data collection manifest, an additional label for the subscription identifier is required. For instance, the same datapoints as in could be stored as in .
Storing datapoints with information to retrieve the data manifest
From the "interfaces_interface_enabled" datapoint, one can retrieve the corresponding data collection manifest by looking for datapoints that have the label data_collections_data_collection_yang_push_collection_subscriptions_subscription_id equal to 4242. Various optimizations could be done, such as relying on on-change subscription to modify only the leaves that changed. In that way, the amount of data needed for updating and storing the data manifest in the TSDB would be limited.
Generating YANG Tree Diagrams This section provides the files needed to generate the YANG tree diagram from . The diagram was obtained using yanglint version 2.1.80, using the YANG Schema Mount . It was manually edited to remove parts irrelevant to this document such as data nodes from imported modules, notifications and RPCs. In order to get a tree diagram involving YANG Schema Mount with yanglint, two data files are required, in addition to the YANG module, its dependencies and the YANG modules to be mounted. The first required file the extension data, containing the YANG library to use at the mount point, this file is provided below as "data-collection-extension-data.xml". The second required file is the YANG library to use at the top-level context, this file is provided below as "data-collection-toplevel-yanglib.xml". The following command was used to obtain the YANG Tree diagram (before manual edition). yanglint -f tree \ -x data-collection-extension-data.xml \ -Y data-collection-toplevel-yanglib.xml \ example-collection-manifest@2025-02-21.yang mountee-set ietf-subscribed-notifications 2019-09-09 urn:ietf:params:xml:ns:yang:ietf-subscribed-notifications xpath ietf-yang-push 2019-09-09 urn:ietf:params:xml:ns:yang:ietf-yang-push on-change ietf-yp-current-period 2025-02-21 urn:ietf:params:xml:ns:yang:ietf-yp-current-period ietf-datastores 2018-02-14 urn:ietf:params:xml:ns:yang:ietf-datastores ietf-yang-library 2019-01-04 urn:ietf:params:xml:ns:yang:ietf-yang-library ietf-inet-types 2013-07-15 urn:ietf:params:xml:ns:yang:ietf-inet-types ietf-interfaces 2018-02-20 urn:ietf:params:xml:ns:yang:ietf-interfaces ietf-ip 2018-02-22 urn:ietf:params:xml:ns:yang:ietf-ip ietf-netconf-acm 2018-02-14 urn:ietf:params:xml:ns:yang:ietf-netconf-acm ietf-network-instance 2019-01-21 urn:ietf:params:xml:ns:yang:ietf-network-instance ietf-restconf 2017-01-26 urn:ietf:params:xml:ns:yang:ietf-restconf ietf-yang-patch 2017-02-22 urn:ietf:params:xml:ns:yang:ietf-yang-patch ietf-yang-types 2023-01-23 urn:ietf:params:xml:ns:yang:ietf-yang-types test-schema mountee-set ds:running test-schema ds:operational test-schema 2 2 example-collection-manifest ]]> main-set ietf-datastores 2018-02-14 urn:ietf:params:xml:ns:yang:ietf-datastores ietf-yang-library 2019-01-04 urn:ietf:params:xml:ns:yang:ietf-yang-library ietf-yang-schema-mount 2019-01-14 urn:ietf:params:xml:ns:yang:ietf-yang-schema-mount example-collection-manifest 2025-02-21 http://example.org/example-collection-manifest ietf-platform-manifest 2025-02-21 urn:ietf:params:xml:ns:yang:ietf-platform-manifest ietf-inet-types 2013-07-15 urn:ietf:params:xml:ns:yang:ietf-inet-types ietf-yang-types 2023-01-23 urn:ietf:params:xml:ns:yang:ietf-yang-types main-schema main-set ds:running main-schema ds:operational main-schema 1 2 ]]>
Validating the Example This section provides the command line for validating the example in using . The files "data-collection-extension-data.xml" and "data-collection-toplevel-yanglib.xml" are provided in the previous section. The file "manifests-example.json" in the one from . yanglint -e -x data-collection-extension-data.xml \ -Y data-collection-toplevel-yanglib.xml \ manifests-example.json
Acknowledgements Thanks to Mohamed Boucadair, Tianran Zhou, Jan Lindblad, Ahmed Elhassany, Joe Clarke, Alex Huang Fang, Zhuoyao Lin and Quifang Ma for their reviews and comments.