]> YANG Message Keys for Message Broker Integration Swisscom
Binzring 17 Zurich 8045 CH thomas.graf@swisscom.com
Swisscom
Binzring 17 Zurich 8045 CH ahmed.elhassany@swisscom.com
INSA-Lyon
Lyon FR alex.huang-feng@insa-lyon.fr
Everything OPS
Liege BE benoit@everything-ops.net
General NMOP YANG-Push Data Mesh Network Telemetry Network Analytics This document specifies a mechanism to define a unique message key for a YANG to message broker integration and a topic addressing scheme based on YANG-Push subscription type and a YANG index defined in this document. This enables YANG data consumption of a subset of subscribed YANG data, either per specific YANG node, identifier or telemetry message type, by indexing and organizing in Message Broker topics, indexing the information the right way.
Introduction Nowadays network operators are using machine and human readable YANG to model their configurations and monitor YANG operational data from their networks according to . Most network analytic use cases require real-time data and deliver near real-time analytical, actionable insights. This imposes high scalability, resilience and low overhead in the data processing pipeline. Accessing the right data for the right use case with minimal overhead and in the shortest period of time is therefore crucial. Network operators organize their data in a Data Mesh according to where a Message Broker such as Apache Kafka or Apache Pulsar facilitates the exchange of messages among data processing components in topics and subjects. Typically, data is being stored in Message Broker topics for several hours or days to facilitate resilience in the data processing chain and addressed in subjects depending on schema, enabling a data consumer to address and re-consume previously consumed data again if previously lost. Dimensional data is structured information in a data store. It uses a model of dimension tables to organize business metrics and their descriptive context. This model, developed by Ralph Kimball, simplifies data analysis and reporting by creating denormalized, easy-to-understand structures for quick querying. It is optimized for online analytical processing (OLAP) and data warehouses. YANG as a data modelling language facilitates the modelling of dimensional data. An Architecture for YANG-Push to Message Broker Integration specifies an architecture for integrating YANG-Push with Message Brokers for a Data Mesh architecture. How the notification messages at a YANG-Push Receiver are being transformed to the Message Broker is described in and to which message schema in , however how messages should be indexed best for dimensional YANG data is left unspecified. Due to the missing dimensional indexing for Message Broker stored YANG data, all YANG data is stored in one single Topic, distributed round robin across multiple Partitions and each YANG schema id is a subject within that topic. Therefore, the entire Topic from all Partitions needs to be consumed first before data selection can be applied. This leads to avoidable data processing overhead which in turn impairs scalability and real-time capabilities, required for certain Network Analytics use cases. YANG telmetry data can be used for several network analytic use cases. Importantly, depending on the use case, only a subset of the subscribed YANG data might be necessary (in time or space). For example, for specific use cases, it's more important to know the current network state, as opposed to have the full series of the state changes over time. In other use cases, instead of consuming data for all network nodes, only a specific network node or network node component requires the YANG monitoring and hence subscription. This document defines how YANG messages should be indexed and organized in Message Broker topics by leveraging the network node hostname, YANG datastore name and YANG Item Identifier for indexing; and YANG-Push subscription type and YANG schema name for a Message Broker topic naming scheme. Network node hostname, YANG datastore name and subtree and xpath filters are part of "ietf-yang-push-telemetry-message" structured YANG data defined in . YANG item identifier are derived from subtree and xpath filters respectively from their YANG schema tree.
Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.
Terminology The following terms are used as defined in :
  • Network Telemetry
  • Network Analytics
  • Value
  • State
  • Change
The following terms are used as defined in :
  • Message Broker
  • YANG Message Broker Producer
  • YANG Message Broker Consumer
The following terms are used as defined in Apache Kafka and Apache Pulsar Message Broker:
  • Subject: A named communication channel where a schema id is associated.
  • Topic: A communication channel for publishing and subscribing messages with one or more subjects.
  • Topic Compaction: The act of compressing messages in a topic to the latest state. As used with Apache Pulsar. Apache Kafka uses the term Log Compaction with identical meaning.
  • Partition: Messages in a topic are spread over hash buckets where a hash bucket refers to a partition.
  • Message: A piece of structured data sent between data processing components to facilitate communication in a distributed system
  • Message Key: Metadata associated with a message to facilitate deterministic hash bucketing.
The following terms are used as defined in Confluent Schema Registry Documentation:
  • Schema: A formalized, documented structure that defines the shape and content of the messages exchange.
  • Schema ID: A unique identifier of a schema associated to a Message Broker subject.
The following terms are used as defined in :
  • Periodical
  • On-Change
  • Sync-On-Start
  • Xpath Filter
  • Subtree Filter
The following terms are used as defined in :
  • Notification
  • Hostname
The following terms are used as defined in :
  • Datastore
The following terms are used as defined in :
  • Schema Node Identifier
  • Schema Tree
The following terms are used as defined in :
  • YANG Item Identifiers
This document defines the following term:
  • YANG Index: Is a subset of YANG Item Identifiers containing only schema node identifiers. Different to an absolute schema node identifier it includes the YANG module name and is therefore globally unique. When the schema node identifier points to a YANG list, then the key to that list is included.
Solution Design In order to identify which YANG node identifier of a network node YANG datastore is produced in which Message Broker Topic and Partition for which Subject, YANG Message Keys and Indexes are being introduced. In order to facilitate Message Broker Topic Compaction, a YANG-Push subscription type based topic naming scheme is proposed. This segregates statistical (Value), State and State change YANG metrics and facilitates a YANG Message Broker Consumer to use the Topic wild card consumption method to select based on YANG-Push subscription type.
YANG Message Keys and Indexes A Message Broker MUST use a Message Key to index the message and a value to carry the Message content. If no Message Key is defined then the Messages are distributed in a round robin fashion across partitions. If a Message Key is defined, then the value of the Message Key is being used as input for the Message Broker Producer hash function to distribute across Partitions. Therefore, Message Keys facilitate Message ordering. The Message Key not only used for Message indexing at the Message Producer but also at the Message Broker for topic compaction. For YANG, the network node hostname, from which YANG datastore the YANG metrics are published from and the YANG index is used to generate the Message Key.
YANG Message Broker Producer YANG data nodes are uniquely identifiable within the YANG schema tree. defines with "absolute-schema-nodeid" how absolute YANG schema node identifiers are being crafted locally unique to the YANG module. defines how globally unique YANG Item Identifiers are defined as text strings. defines how YANG data nodes can be subscribed with subtree and xpath selection filters. A YANG-Push publisher publishes with "subscription-started" state notifications for each subscription which filter and filter type is being used to the YANG-Push receiver. To calculate the YANG Index of the Message Key, the YANG item identifier needs to be extracted from the used YANG-Push subtree or xpath subscription filter. If the YANG item identifier is a YANG list as defined in the YANG list key defined in statement is suffixed with a "/" to the YANG Item Identifier. For example, if the following xpath filter is being used, the YANG Item Identifier is "ietf-interface:interfaces/interface". Interface is a YANG list with name as key. Therefore, the YANG Index of the Message Key is "ietf-interface:interfaces/interface/name".
For example, if the following subtree filter is being used, the YANG Item Identifier is "ietf-hardware:hardware/component/state". Therefore, the YANG Index of the Message Key is "ietf-hardware:hardware/component/state".
]]>
When the Message is being produced to the Message Broker, the Network node hostname and YANG datastore name is used from the structured YANG data defined in "ietf-yang-push-telemetry-message" where the YANG Index is derived from subtree and xpath filters, respectively from their YANG schema tree.
YANG Message Broker Consumer The consumer hashes the Message Key and applies modulo with the number of partitions to determine the partition it needs to consume from to obtain Messages with desired Message Key.
YANG-Push Message Broker Topic Naming YANG can be subscribed periodically, on-change or on-change with sync-on-start. Periodical subscriptions are used for obtaining statistical metrics. On-Change subscriptions are used for obtaining State Changes and on-change with sync-on-start for obtaining States. Message Brokers topics are addressed with a unique name. Usually topics are named hierarchically similar to the DNS namespace where "." deliminates hierarchies. This document defines "statistics", "states" and "state-changes" in the topic name as the first part to denote the types of data. Followed by "yang" to denote YANG data. Followed by the YANG module names subscribed, and followed by the YANG Schema Node Identifier where "/" is substituted by "_". For example, if the "ietf-interface:interfaces/interface" xpath filter is being used, the Message Broker topic name would be as following. In the example the project name and environment (prod, dev, test etc.) is prefixed.
YANG Message Broker Producer For the Message Broker topic creation, the "periodic", "on-change" and "sync-on-start" contained data in "update-trigger" from "ietf-subscribed-notifications", YANG module defined in , subscription state notifications MUST be used to derive wherever subscribed YANG data is "statistics", "states" or "state-changes". The YANG Index MUST be derived from subtree and xpath filter data of subscription state notifications, respectively from their YANG schema tree.
YANG Message Broker Consumer The consumer has the ability to consume with a wildcard denoted with "*" in the topic name to consume from more than one topic. For example, if YANG states should be consumed and indexed in Time Series database or stream processor than below Topic Name could be used, and the YANG data could be ingested into tables according to topic names and indexed per Message Key. If Topic Compaction is enabled, only current state is consumed.
Message Broker Implementations Topic, Partitioning and Message Keying are generic concepts of Message Brokers. There are two known Message Broker implementations supporting all features described in this document.
Apache Kafka Apache Kafka supports Message Keying, Partitioning and Log Compaction. The topic names are constrained to 249 character length and the following characters: "a-z", "A-Z", "0-9", ".", "_" and "-".
Apache Pulsar Apache Pulsar supports Message Keying, Partitioning and Topic Compaction. The topic names allow all characters except: "/".
IANA Considerations This document includes no request to IANA.
Security Considerations This document should not affect the security of the Internet.
Operational Considerations The YANG Message Broker Producer of a YANG-Push receiver should have three config knobs facilitate the features described in this document as optional:
  • Topic Distribution: Select between "topic" and "subject" distribution. Default is subject to remain backward compatibility to .
  • Distribution Type: Select between "none" and "YANG-Push subscription type".
  • YANG Message Key: Select between "enable" and "disable".
To accommodate for potential date loss throughout the data processing pipeline, periodical update of the current State for State metrics is RECOMMENDED. This can be accommodated with YANG-Push as defined in by complementing "on-change sync on start" subscriptions with periodical subscriptions. Alternatively, in YANG-Push Lite defined in this simplified in one subscription.
References Normative References Informative References Toward Building a Semantic Network Inventory for Model-Driven Telemetry IEEE Toward Avoiding the Data Mess: Industry Insights From Data Mesh Implementations IEEE Data Mesh O'Reilly Media The Data Warehouse Toolkit Wiley Apache Kafka Apache Software Foundation Apache Pulsar Apache Software Foundation Confluent Schema Registry Documentation Confluent Community and Apache Software Foundation
Acknowledgements Thanks to
Contributors We like to thank Victor Lopez for the initial idea on the network controller use case. Ashley Woods, Sivakumar Sundaravadivel and Rafael Julio for the idea of grouping topics by YANG-Push subscription type and insisting that Topic Compaction is a key enabler for inventory metrics and YANG data consumer integration and should be supported day 1. And Nigel Davis for confirming that Topic Compaction simplifies indeed data processing system architecture.