SenML is CORECONF (almost)

Introduction In its simplest form, an IoT device consists of at least one sensor, and ability to send measurements of this sensor over the network. Occasionally, the device parameters can also be sent over the network to monitor and manipulate its behavior. Such devices are constrained on energy, network (in its availability as well as bandwidth), and data processing capabilities as they embed low power processors. Consequently, SenML is an appropriate choice for representing this nature of data from these devices. As much as SenML is useful in building simple IoT applications, a well-defined data model for the same would allow developers and engineers alike to build more complex data systems if the data can be modeled in YANG. Subsequently, the YANG model of SenML can leverage SID based CORECONF representation of its data to further reduce network footprint and improve its interoperability with other network devices.

SenML Format SenML or Sensor Measurement Lists is an data format used by the constrained devices to send sensor information over the networks . These measurements are often structured as key-value pairs where the keys (also known as fields) describe the associated sensor data. Each field has a well defined label, whether it is mandatory to be included and the permitted values it can carry. SenML also reduces sending redundant information over the network by introducing concepts such as base-name, base-unit, base-version, base-time and base-value. The format specifies how the application payload can be serialized in three popular formats before sending it over the network- JSON (JavaScript Object Notation) , CBOR (Concise Binary Object Representation) and XML (Extensible Markup Language). Although a SenML record has a well-defined Concise Data Definition Language (CDDL) for JSON and CBOR representations in section 11 of {RFC8428}}, the lack of an accompanying data model means it is harder to use it for applications with strict requirement for data organization and validation. Additionally, SenML CDDL cannot be used directly to extend to other data formats such as CoAP Management Interface (CORECONF).

YANG Model Language YANG or Yet Another Next Generation is data modeling language used to describe the organization and constraints on the configuration and state data of the network devices. This data is typically exchanged using NETCONF or RESTCONF protocols . As YANG models are considered de-facto interchange formats for a particular protocol (or application) which allows network device manufacturers to build inter-operable devices. YANG has rich data types, language features, and supports several extensions for constructing user-defined data types, recursive data models and allowing inheritance to reuse existing data models among others. Logically, to model SenML format into a YANG model, the measurements can be designed as YANG lists and each SenML record is a grouping containing leaves of SenML fields and values . Additional constraints and rules can be added to the model ensuring conformance with SenML specification. Visually it can be represented as follows: SenML Element YANG Equivalent Measurement List Field Leaf Label Leaf Name Value Type Leaf Type Record Grouping/Container Section 11 of specifies how SenML format is described for encoding into JSON, XML and CBOR. However, IoT devices which use YANG data models don't have any easy way to incorporate and validate their SenML measurements. Additionally, in absence of a YANG model, low powered devices which send measurements in SenML, cannot use CORECONF representation model without resorting to some sort of intermediate data processing. Hence, if SenML has a YANG model, these low powered devices can describe their data entirely in it. Their data can further be transformed in CORECONF, which complies with SenML-CBOR encoding rules before transmitting it over the network. To enable accurate, fast and efficient transmission of data conforming to YANG models, Schema Identifiers (known as SIDs) can be generated and assigned for each YANG element . These SIDs can be used to transform data in CORECONF format as demonstrated in . An example of this is also described in Transforming SenML to CORECONF below.

YANG Model for SenML: Considerations As described in , SenML measurements consists of field and a value, and each field is identified by its label (which are different for JSON and CBOR). Each value has a well-defined data-type for encoding in JSON, in CBOR or in XML as outlined in section 12.2 RFC 8428. Hence, to describe a generic SenML model in YANG, it is necessary to represent each field as a YANG leaf with the most appropriate YANG type associated with the type from the "XML Type" Column. A user data type is created in the YANG model to make XML Integer like type-definition. XML Double type follows IEEE 754-2008 definition, which results in approximately 15 significant digits. However for the YANG model, XML Double is replaced by a Number type, which is a decimal64 type with 5 precision digits and has a range of [-92233720368547.75808, 92233720368547.75807]. This is chosen arbitrarily to balance precision and range in YANG but can be changed by the user later. bn:
Base Name, which is directly mapped to String bt:
Base Time, which is mapped to closest type available in YANG, decimal64. bu:
Base Unit, which can be mapped to string type in YANG with special rules where units can be restricted to ones listed in Section 12.1 (primary units) of or can be extended to support secondary units described Section 3 in . bv:
Base Value, which can be mapped to decimal64 type in YANG, with precision of 5 digits. This can be overridden using redefine keyword when this YANG module is imported. bs:
Base Sum, live Base Value, can be mapped to decimal64 type and its precision can be overridden later. bver:
Base Version, which is an integer can be modeled union type to assume either as a int8, int16, int32 or int64 types. n:
Name, modeled as string. u:
Unit, modeled as string. leaves v,vs,vb,vd:
Value, Value String, Value Boolean and Value Data, modeled as a choice type as exactly one field must appear unless there is a sum field, in which case it is allowed to have no value fields. s:
Sum, mapped as a decimal64 with 5 digit precision. t:
Time, mapped as a decimal64 with 5 digit precision. Although it is optional in SenML to provide time but for this YANG model, it is mandatory-leaf if measurement lists are used as time is the key for the measurements list (e). ut:
Update Time, mapped as a decimal64 with 5 digit precision. list e:
SenML measurement lists, labeled here as "e" (events) is mapped as a list type in YANG with constraint on time (t) as the key. Thus, t has to be present and unique in the measurement lists.

YANG Model for SenML: Implementation

file "senml.yang" module senml { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:senml"; prefix sen; organization "LWM2M Working Group"; typedef Integer { type union{ type int8; type int16; type int32; type int64; } } typedef Number { type decimal64{ fraction-digits "5"; } } grouping senml-grouping { leaf bn { type string; } leaf bt { type Number; } leaf bu { type string; must "(. = 'm' or . = 'kg' or . = 'g' or . = 's' or . = 'A' or . = 'K' or . = 'cd' or . = 'mol' or . = 'Hz' or . = 'rad' or . = 'sr' or . = 'N' or . = 'Pa' or . = 'J' or . = 'W' or . = 'C' or . = 'V' or . = 'F' or . = 'Ohm' or . = 'S' or . = 'Wb' or . = 'T' or . = 'H' or . = 'Cel' or . = 'lm' or . = 'lx' or . = 'Bq' or . = 'Gy' or . = 'Sv' or . = 'kat' or . = 'm2' or . = 'm3' or . = 'l' or . = 'm/s' or . = 'm/s2' or . = 'm3/s' or . = 'l/s' or . = 'cd/m2' or . = 'W/m2' or . = 'bit' or . = 'bit/s' or . = 'lat' or . = 'lon' or . = 'pH' or . = 'dB' or . = 'dBW' or . = 'Bspl' or . = 'count' or . = '/' or . = '%' or . = '%RH' or . = '%EL' or . = 'EL' or . = '1/s' or . = '1/min' or . = 'beat/min' or . = 'beats' or . = 'S/m' )" { description "The unit must be from the approved unit list"; error-message "Invalid unit. Unit must be one from SenML Units Registry"; } } leaf bv { type Number; } leaf bs { type Number; } leaf bver { type Integer; } leaf n { type string; } leaf u { type string; } choice valueleaf { description "Exactly one Value field MUST appear unless there is a Sum field, in which case it is allowed to have no Value field"; leaf v { type Number; } leaf vs { type string; } leaf vb { type boolean; } leaf vd { type string; } } leaf s { type Number; } leaf t { type Number; } leaf ut { type Number; } } list e { key t; uses senml-grouping; min-elements 0; max-elements unbounded; } }

]]>



The pyang tool can be used to visualize the YANG tree:



An example instance of JSON data instance for the above YANG model is presented below:



It is possible to validate this YANG model against a SenML data in json encoding as shown below using yangson library:

 file "validate_senml_yang.py"
import json
import yangson

from yangson import DataModel as DM
from yangson.enumerations import ContentType

# This function is required to convert all numerical values to strings due to yangson's strict typing 
def convert_numbers_to_strings(d):
    for key, value in d.items():
        if isinstance(value, (int, float)):
            d[key] = str(float(value))
        elif isinstance(value, dict):
            convert_numbers_to_strings(value)  # Recursive call for nested dictionaries
        elif isinstance(value, list):
            for item in value:
                convert_numbers_to_strings(item)
    return d

# Create a data model from model definition in JSON format and path to YANG modules
dm = DM.from_file("./yang-library-senml.json", ["./"])

# Load the SenML data in JSON format
jsonData = json.load(open("senml_example.json"))

jsonData = convert_numbers_to_strings(jsonData)
rootInstanceNode = dm.from_raw(jsonData)

# Validate the root instance node
try:
    rootInstanceNode.validate(ctype=ContentType.all)
    print("Validation successful")
except:
    print("Validation failed")

]]>


Transforming SenML to CORECONF

Next, for the above YANG model, we can generate the SIDs using pyang tool starting from integer 60000, as follows:



However, we can modify the SIDs manually so that the resultant CORECONF will have deltas conforming to SenML-CBOR Labels as defined in section 12.2 of :



SID allocation rule states that for every unique YANG identifier, there is a corresponding unique SID number. This rule is broken here to ensure the delta for "n" is computed to be 0 to match SenML-CBOR label definition.
Hence, top-level key "e" and "n" have the same SID. However, "e" is not part of existing SenML label set as defined in the section 4.3 of .

The resulting CORECONF diagnostic format of the data instance is shown below:



Corresponding CBOR encoded hexadecimal is shown below:



If the module name contained in the first 4 bytes ("a119ea60") is removed, then the resultant CBOR conforms to SenML CBOR specification and can be easily parsed by SenML parsers.

Thus, a SenML CBOR representation can be almost CORECONF representation just by adding 4 bytes of the encoded module name.

Finally, a simple comparison of the JSON with the corresponding CORECONF-CBOR encoded form is shown below:


      4 byte module-name
      JSON Size
      CORECONF Size
      Compression %
      Present
      136 Bytes
      104 Bytes
      23.53
      Absent
      125 Bytes
      100 Bytes
      19.99



Further work

Some areas of future work have been identified to improve SenML YANG modelling-


  Base units can be either modelled as enumerated type or identityref type, so they can be assigned a SID value. This should ideally further reduce the CORECONF-CBOR encoded message.
  Constraints on having leaves v, vs, vb, vd optional if s is available as described in Section 4.2 of 
  Add secondary units described in  in the base units constraints.
  Uniqueness for SIDs can be restored by modifying SenML-CBOR label value for "n" to be any unused integer except 0.