SID Extension to efficiently manipulate YANG Data Models

Introduction The YANG modeling language is very popular for node configuration and retrieving the the state of a device.
XML and JSON are also two popular formats to serialize data in conformance with the data model. Recently, a new serialization format has been published, allowing a more compact representation of the information. This new format is based on CBOR and the YANG identifier; instead of being represented by a unique ASCII string, values are mapped to an unique integer. The mapping between strings and integer is kept in a .sid file usually generated by tools such as pyang. This document presents some extensions to the sid files that allow to ease the conversion between CBOR and JSON, as well as manipulating YANG Data Models in a constraint environment.

YANG Data model YANG is a modeling language to structure information and check its conformity. Each element of a YANG Data Model is identified through an unique identifier. YANG is based on a hierarchical approach. During the serialization phase, data is represented either in XML or JSON. In this document we will use the JSON representation. indicates how to form a JSON structure conforming to a YANG Data Model: Leaves are represented as JSON objects, the key being the leaf name. Lists are defined through arrays. Identityref is a string containing the identifier in the YANG naming hierarchy for an identity. The YANG Data Model, in , is used to illustrate how data will be serialized. It defines a container representing a physical object able to perform several measurements. The physical object is battery powered, has a status LED able to change color, and a list of attached sensors returning an integer value.

file "sensor.yang" module sensor { yang-version 1.1; namespace "http://sensorexample.in/sensor-example"; prefix sen1; identity battery-indicator-base-type { description "a base identity for battery level indicator"; } identity high-level { base battery-indicator-base-type; } identity med-level { base battery-indicator-base-type; } identity low-level { base battery-indicator-base-type; } typedef battery-level { type identityref { base battery-indicator-base-type; } } container sensorObject { leaf statusLED { type enumeration { enum green {value 0;} enum yellow {value 1;} enum red {value 2;} } } leaf battery { type battery-level; } list sensorReadings { key "index"; leaf index { type uint8; } leaf sensorValue { type uint32; } } } }

]]>



The YANG tree regarding this Data Model is given in . The tree displays the module hierarchy. For the module "sensor", a container "sensorObject" contains two leaves ("statusLED", "battery") and a list ("sensorReadings").


JSON serialization

An example of data, serialized with JSON, and conforming to the YANG Data Model, is given in . Embedded JSON Objects allow 
to represent the hierarchy. The key of the outer JSON Object is composed of the module name and the container name. The embedded JSON Object associated to this  contains the leaves as keys. The YANG list is represented by an JSON Array.




CBOR Serialization

JSON notation is verbose for constrained networks and objects. To optimize the size of the representation,  defines a CBOR serialization, also used in CORECONF. YANG ASCII identifiers are replaced by unique number. In JSON, the uniqueness is guaranteed by the "namespace" URI, as shown in . By construction, the rest of the identifiers are unique.

In CORECONF, the uniqueness is guaranteed through the use of positive integers called SID, which replace the ASCII identifiers.  defines the allocation process. Module developers may ask for a SID range from their authority. For example, for an IETF module, the IANA will allocate a SID range.

The  shows an example of this conversion. The range is arbitrarily fixed between 60000 and 60099. Note that the module, the identities, and the leaves have an assigned SID.



To perform the allocation, the pyang utility generates a .sid file in the JSON format, resulting in a YANG Data Model specified in . The  gives an excerpt.



The serialization in CBOR of the JSON example in  is given in . Compared to the compacted representation of the JSON structure (152 Bytes), the CORECONF structure is 24 bytes long. Although the size is different, the structure remains the same. It is composed of embedded CBOR Maps (equivalent of JSON Object). The first key is a SID (60005 corresponds to the container). Embedded CBOR Maps use a delta notation to encode the keys. The key 5 corresponds to SID 60005+5, thus to the leaf "statusLED". Key 2 in the second CBOR Map corresponds to the YANG list "sensorReadings", and the elements of the list are stored in a CBOR Array.

Note that in this example, the enum for "statusLED" (60005+5) is an integer and the identityref for "battery" (60005+1) is also an integer pointing to the  SID "med-level" (60004).




Conversion between JSON and CBOR

Even if the conversion between CBOR and JSON formats might look obvious, including data compatible with a YANG Data Model is not so trivial.
The reason is that JSON uses ASCII identifiers for readability and CBOR prefers integers for compactness. The  summarizes some YANG types when coded in JSON () or CBOR ().


      YANG
      JSON
      CBOR
      int8, int16, int32, uint8, uint16, uint32
      number
      +int, -int
      int64, uint64
      string
      +int, -int
      decimal64
      string
      CBOR Tag 4
      binary
      base64
      byte array
      bits
      string
      array
      boolean
      boolean
      boolean
      identityref
      string
      +int
      enumeration
      string
      +int


The conversion from CBOR to JSON is not direct, since the YANG type needs to be considered. For instance, a integer could be converted into a number, an enum, or an identityref. Note that for union, the conversion is simple since some CBOR Tags may be used to indicate how the integer is converted, but outside of the union, for a single value, there is no such clue.

On the other direction, a JSON string may correspond to a 64-bit long number, an enumeration, or an identityref.

To perform the conversion, the YANG type is needed, but popular tools such as yanglint or pyang are not currently manipulating CBOR representation. Furthermore, these tools cannot run on constrained devices. To overcome these problems, we propose to extend the .sid file with more information. The modified pyang code (see https://github.com/ltn22/pyang/tree/sid-extension) adds useful information to the .sid file when the "--sid-extension" argument is provided.

This extension contains a "type" key in the JSON Object describing the mapping between the SID and the identifier if the node is a leaf (see ).



The "type" key added to leaf nodes contains several information:


  when the "type" key is not present, the node is a not leaf and there is no need for type conversion.
  when the value of "type" is a string, it gives the YANG type of the leaf, as defined in the YANG Data Model. If the YANG type derives from an identityref, the value "identityref" is given instead of the YANG type specified in the module. In that case, in the CBOR notation, there is a pair "{ ...., SID1: value1, ...} in the CBOR Map. A first lookup at the .sid file for SID1 returns that the type is "identityref", a second lookup for value1 in the "identity" namespace, returns the ASCII identifier.
  when the value of "type" is a JSON Object, then the leaf is a YANG enumeration, and the CBOR Map gives the mapping between integer and strings.
  when the value of "type" is a JSON Array, then the YANG leaf is a union. Elements of the Array list the possible types. In that case, the conversion is leaded by the CBOR Tags associated to the value.



We developed the pycoreconf Python module to facilitate the conversion between JSON and CBOR (https://github.com/ltn22/pycoreconf). The  gives an example of a Python script using this module. It takes as input the .sid file and a JSON structure. It transforms it into a CBOR structure, and back to JSON representation.



The resulting JSON structure of  is given in . It shows that the JSON format can be converted to CBOR and vice versa, just by using the extended .sid file.




Navigating the CORECONF structure.

As we saw, the CORECONF data  is structured as a tree. The  gives an example for the example module described in . The numbers are the SID associated to the YANG Data Model.  defines how queries are made on this structure to get the full tree or a sub tree.



Requesting the SID corresponding to the module (60005), will return the full tree. Requesting "battery" leaf (60006) will return the leaf and the associated value, while requesting "sensorReadings" (60007) will return the subtree with all its values. Going deeper in this sub-tree imposes the use of keys. For instance, to get a specific "sensorValue" one must provide the key "index" (60008).

In , the series of keys needed to reach a specific element is provided either in the query string or in the fetch. Only one list is provided in , but several lists can be embedded, and each level may require one or more keys. So, to be processed, the YANG Data Model as to be known.

To avoid parsing the YANG Data Model on a constrained device, we propose to summarize this information into a JSON Object, generated by pyang with the "--sid-extension" argument.



The "key-mapping" key points out to a JSON Object, where the JSON key is a SID corresponding to a YANG list key, and the value is a JSON array containing the SIDs acting as a YANG key for that YANG list. The "key-mapping" JSON Object indicates:


  that a SID is a YANG list, since its value appears as a JSON key in the "key-mapping" structure.
  the number of elements in the corresponding array indicates how many items are involved as a YANG list key. The value for these items can be taken from the request.
  the array also indicates which SIDs contain the value that must be compared to the YANG list key sent in the request.


With the information contained in the "key-mapping" structure, it is possible to browse any CORECONF structure and return the appropriate requested SID.


Linking to real values

In some YANG Data Models, as the one provided in , some leaf nodes might encompass information that extends beyond the model's boundaries.
For example, the "statusLED" leaf may require an action to be triggered when a value is written onto it, or the managed device might need to interact with a physical sensor when the "sensorValue" leaf is queried.
On the other hand, certain leaf nodes, such as "index", do not require any interaction as their values are directly stored in the database.

We wrote a script taking the .sid file obtained from pyang with the "--sid-extension" argument to produce template C code for these functions. We try keep SID transparent and use the YANG identifiers as much as possible, which are easier to manipulate by a programmer.

The tool aims to generate aliases for identifiers that also include details about their associated SIDs. This eliminates the need for developers to recall or verify SIDs during software development.


  The tool is hosted at https://github.com/manojgudi/ccoreconf/tree/stub_generation/tools
  For the SID file above (sensor@unknown.sid), the second argument in the command- "sensor_prototypes" instructs the tool to create two files sensor_prototypes.h and sensor_prototypes.c for writing headers and function code, respectively. To generate the C code, simply run the tool:





#include 
#include 
#include 
#include 

#define  SID_BATTERY             60006
#define  SID_INDEX               60008
#define  SID_SENSORVALUE         60009
#define  SID_STATUSLED           60010


#define read_sensorObject        read_60005
#define read_battery             read_60006
#define read_sensorReadings      read_60007
#define read_sensorValue         read_60009
#define read_statusLED           read_60010

char* keyMapping = "\xa1\x19\xeag\x81\x19\xeah";
enum StatusledEnum {green = 0, yellow = 1, red = 2};

void  read_sensorObject(void);
char *  read_battery(void);
void  read_sensorReadings(uint8_t index);
uint32_t  read_sensorValue(uint8_t index);
enum StatusledEnum read_statusLED(void);
]]>


#include 
#include 
#include 
#include "sensor_prototypes.h"

/*
    This is an autogenerated function associated to 
    SID: 60005
    Module: data 
    Identifier: /sensor:sensorObject
    function params: 
    Stable: false
*/
void  read_sensorObject(void){
    // Initialize the leaf if it has a return type with a default value;
    // Do something with the leaf
}

/*
    This is an autogenerated function associated to 
    SID: 60006
    Module: data 
    Identifier: /sensor:sensorObject/battery
    function params: 
    Stable: false
*/
char *  read_battery(void){
    // Initialize the leaf if it has a return type with a default value;
    char * batteryInstance  = NULL;

    // Do something with the leaf
    // Return the leaf
    return batteryInstance;
}

/*
    This is an autogenerated function associated to 
    SID: 60007
    Module: data
    Identifier: /sensor:sensorObject/sensorReadings
    function params: /sensor:sensorObject/sensorReadings/index
    Stable: false
*/
void  read_sensorReadings(uint8_t index){
    // Initialize the leaf if it has a return type with a default value;
    // Do something with the leaf
}

/*
    This is an autogenerated function associated to 
    SID: 60009
    Module: data 
    Identifier: /sensor:sensorObject/sensorReadings/sensorValue
    function params: /sensor:sensorObject/sensorReadings/index
    Stable: false
*/
uint32_t  read_sensorValue(uint8_t index){
    // Initialize the leaf if it has a return type with a default value;
    uint32_t sensorValueInstance  = 0;

    // Do something with the leaf
    // Return the leaf 
    return sensorValueInstance;
}

/*
    This is an autogenerated function associated to 
    SID: 60010
    Module: data 
    Identifier: /sensor:sensorObject/statusLED
    function params: 
    Stable: false
*/
enum StatusledEnum read_statusLED(void){
    // Initialize the leaf if it has a return type with a default value;
    enum StatusledEnum statusLEDInstance;

    // Do something with the leaf
    // Return the leaf 
    return statusLEDInstance;
}
]]>


Understanding Stubs

Let us take a quick look at the generated code snippet to understand how they help abstract SIDs from the developers:

The first part of the sensor_prototypes.h contains pre-processor directives which basically store the identifiers with their corresponding SID values for quick access. The second part contains function prototypes which can be implemented later. Note that there will not be any function prototype generated to read SID keys (in this case index, with SID 60008).



Now let us understand what is generated in sid_prototypes.c by looking at the auto-generated function prototype read_sensorValue. The program auto-infers that the leaf sensorValue can only be reached through a valid SID key (that is, index), so a function trying to read sensorValue from its CORECONF instance will require a parameter index. Also, since sensorValue has a well-defined type in C, the function will try to return that uint32_t.

The program also maps the enum type from SID to a corresponding enum type in C, as shown for the leaf statusLED. The value of these enums is as specified in the corresponding YANG model. If unspecified, the values are auto-assigned.

For rest of the non-leaf functions and leaf nodes with type identityref, the program infers it to be a char * (since the types are non-standard).