Skip to content

N
naiades-platform-poc

Pushing Data from pilots to IoT Platform

Last edited by Federico Sismondi
Page history

This document explains how to push data from City Pilots to IoT Platform. Basically it covers 3 things:

  1. How we can use Device data model for IoT devices in pilots.
  2. How to push real-time Device data to IoT Platform.
  3. How to push locally collected (e.g. scada or data loggers) Device data to IoT Platform.
  4. How to push all locally collected as a batch operation

Before starting to describe Device data model, you should ask yourself:

Should I use Device data model or a more specific one related to the domain of use:

This is a very important question to ask yourself before choosing to use Device data mode.

DCA can choose to use a more specific data model for pushing the data they get from IoT devices, e.g. weather environmental stations IoT device can/should push data using WeatherObserved data model. This is probably a more convenient option regarding the posterior data usage (consumers of data). In other words consumers of data will appreciate you using domain specif data models (such as WeatherOberved) rather than using more generic models (such as Device). Device data models should be used when handling large heterogeneous data sets, which dont fit well any of the well-known data models.

1. Using Device Data model

Some pilots data collectors (DCA) have expressed the need of pushing measurements coming from IoT devices or from SCADA systems.

The Device Data model is documented here:

https://fiware-datamodels.readthedocs.io/en/latest/Device/Device/doc/spec/index.html

Fitting pilot requirements to data model

  1. Pilots require to model devices pushing single measurements, e.g. an IoT temperature sensor
  2. Pilots require to model devices pushing multiple measurements, e.g. an IoT temperature and soil_moisture sensor

For this, the proposal is to use create a Device entity in the following way:


   {
        "id": "urn:ngsi-ld:Device:Device-test",
        "type": "Device",
        "value": {
            "type": "Array",
            "value":[17.5,30]
        },
        "controlledProperty": {
            "type": "Array",
            "value": ["temperature","soil_moisture"]
        },
        "location": {
            "type": "geo:json",
            "value": {
                "type": "Point",
                "coordinates": [
                    0.0,
                    0.0
                ]
            }
        }
    }

The data is contained in an array, using the attribute called value. The attribute controlledPropoerty, which is also an array describes the measurands of value. The attribute location describes the location of the sensor using gps coordinates.

Discussion

This model is simple, and flexible, as we can include any sensed data we want in the device entity. On the other side, there is an added complexity for the consumers, and they dont know which attributes are included in an entity until they got fetch the data. This is not the case for example for a WeatherObserved entity, as we always know that these kind of entityes always have an attribute describing e.g. the temperature. To overcome this added complexity this document describes how to use the NGSIv2 powerful querying mechanisms and getting the data you want efficiently, without having to do too much filtering.

Creating the entity in the IoT platform

As an example, you can create an entity using bash:

curl --location --request POST 'http://5.53.108.182:1026/v2/entities/' \
--header 'Content-Type: application/json' \
--header 'Fiware-Service: carouge' \
--data-raw '   {
        "id": "urn:ngsi-ld:Device:Device-test",
        "type": "Device",
        "value": {
            "type": "Array",
            "value":[17.5,30]
        },
        "controlledProperty": {
            "type": "Array",
            "value": ["temperature","soil_moisture"]
        },
        "location": {
            "type": "geo:json",
            "value": {
                "type": "Point",
                "coordinates": [
                    0.0,
                    0.0
                ]
            }
        }
    }
'

Consumers of Device entity

If you know specifically which device you want data from:

curl --location --request GET 'http://5.53.108.182:1026/v2/entities/urn:ngsi-ld:Device:Device-test' \
--header 'Fiware-Service: carouge'

When trying to get all devices in carouge which sense soil_moisture, you can:

curl --location --request GET 'http://5.53.108.182:1026/v2/entities?type=Device&q=controlledProperty==soil_moisture&limit=100' \
--header 'Fiware-Service: carouge'

in this case q=controlledProperty==soil_moisture provides us with the filtering features we want. Other attributes could be used for extending the filtering features.

If you want to know how to get the historical values of a certain value please keep reading.

2. How to push real-time Device data to IoT Platform.

This described how to push data using Device model into the CONTEXT MANAGER (real-time data).

If you read section 1. you know there is an entity called urn:ngsi-ld:Device:Device-test in the IoT Platform. Now suppose we have collected a new value for temperature and soil_moisture, and hence we want to update those values. We may want to update a certain attribute temperature, or soil_moisture, or both. For this we could simply update the Device.value like this:

curl --location --request PUT 'http://5.53.108.182:1026/v2/entities/urn:ngsi-ld:Device:Device-test/attrs/value' \
--header 'Fiware-Service: carouge' \
--header 'Content-Type: application/json' \
--data-raw '{
    "type": "Array",
    "value": [
        40.0,
        122
    ]
}
'

now if you consume again this entity data, you will get:

{
    "id": "urn:ngsi-ld:Device:Device-test",
    "type": "Device",
    "controlledProperty": {
        "type": "Array",
        "value": [
            "temperature",
            "soil_moisture"
        ],
        "metadata": {}
    },
    "location": {
        "type": "geo:json",
        "value": {
            "type": "Point",
            "coordinates": [
                0,
                0
            ]
        },
        "metadata": {}
    },
    "value": {
        "type": "Array",
        "value": [
            40,
            122
        ],
        "metadata": {}
    }
}

Et voilà! Your data is updated inside the context manager of the platform

3. How to push locally collected (e.g. scada or data loggers) Device data to IoT Platform.

This described how to push data using Device model into the HISTORIC DATA component (time series data).

You data is not real-time data, as you have collected it locally, and some time passed between that and now. Also, you probably have more than one set of data, e.g. you may have collected data during the last 24 hours or more, taking samples every 15 mins.

The full documentation for this is at: https://gitlab.distantaccess.com/naiades/naiades-platform-poc/-/wikis/rewriting-history

The API to be used for pushing this data, lives in another component than the CONTEXT MANAGER, as this one only handles real-time data. For this case we will use the HISTORIC DATA component API.

Before starting to push Device data into API we need to check that device exists in the platform, see Creating the entity in the IoT platform section.

Now, image that you have a data logger that logs temperature and soil_moisture , you have collected some sets and you want to make this data available through the IoT Platform.

Here below we show how to POST A SINGLE ENTRY OF THE SET for the date "2020-10-21T16:16:16.00Z", this is:


curl --location --request POST 'http://5.53.108.182:8668/v2/notify' \
--header 'Fiware-Service: carouge' \
--header 'Fiware-ServicePath: /' \
--header 'Content-Type: application/json' \
--data-raw '{
    "subscriptionId": "5f8f0225da7e7c9ac9757a77",
    "data": [
        {
            "id": "urn:ngsi-ld:Device:Device-test",
            "type": "Device",
            "value": {
                "type": "Array",
                "value": [
                    202.01,
                    1992
                ],
                "metadata": {
                    "dateModified": {
                        "type": "DateTime",
                        "value": "2020-10-21T16:16:16.00Z"
                    }
                }
            },
            "controlledProperty": {
                "type": "Array",
                "value": [
                    "soil_moisture",
                    "temperature"
                ],
                "metadata": {
                    "dateModified": {
                        "type": "DateTime",
                        "value": "2020-10-21T16:16:16.00Z"
                    }
                }
            }
        }
    ]
}'

The date when the measurment was taken needs to be formatted as:

"metadata": {
   "dateModified": {
     "type": "DateTime",
     "value": "2020-10-21T16:16:16.00Z"
   }
}

NOTE WELL: All fields of the data model need to include this metadata, else, the current time will be used.

4. How to push all locally collected as a batch operation

If you have plenty of records and not just a single one you can add those to the data of the HTTP message, e.g. image you have locally collected data each hour.

In the following example we show how to push two registers, one generated at "2020-10-21T16:16:16.00Z" and the other generates at "2020-10-21T17:16:16.00Z". The same thing can be done for any amount of registers, although the amount of data that can be received at the same time is limited by a couple of mega bytes. We recommend not pushing more than 1000 registers at a time.


curl --location --request POST 'http://5.53.108.182:8668/v2/notify' \
--header 'Fiware-Service: carouge' \
--header 'Fiware-ServicePath: /' \
--header 'Content-Type: application/json' \
--data-raw '{
    "subscriptionId": "5f8f0225da7e7c9ac9757a77",
    "data": [
        {
            "id": "urn:ngsi-ld:Device:Device-test",
            "type": "Device",
            "value": {
                "type": "Array",
                "value": [
                    202.01,
                    1992
                ],
                "metadata": {
                    "dateModified": {
                        "type": "DateTime",
                        "value": "2020-10-21T16:16:16.00Z"
                    }
                }
            },
            "controlledProperty": {
                "type": "Array",
                "value": [
                    "soil_moisture",
                    "temperature"
                ],
                "metadata": {
                    "dateModified": {
                        "type": "DateTime",
                        "value": "2020-10-21T16:16:16.00Z"
                    }
                }
            }
        },

        {
            "id": "urn:ngsi-ld:Device:Device-test",
            "type": "Device",
            "value": {
                "type": "Array",
                "value": [
                    303.01,
                    3000
                ],
                "metadata": {
                    "dateModified": {
                        "type": "DateTime",
                        "value": "2020-10-21T17:16:16.00Z"
                    }
                }
            },
            "controlledProperty": {
                "type": "Array",
                "value": [
                    "soil_moisture",
                    "temperature"
                ],
                "metadata": {
                    "dateModified": {
                        "type": "DateTime",
                        "value": "2020-10-21T17:16:16.00Z"
                    }
                }
            }
        },

    ]
}'

Getting all historical values of the Device

You can get all historical attributes with:

curl --location --request GET 'http://5.53.108.182:8668/v2/entities/urn:ngsi-ld:Device:Device-test/attrs/value' \
--header 'Fiware-Service: carouge' \
--header 'Fiware-ServicePath: /'

This returns:


{
    "attrName": "value",
    "entityId": "urn:ngsi-ld:Device:Device-test",
    "index": [
        "2020-10-21T16:16:16.000",
        "2020-11-12T11:41:43.000",
        "2020-11-12T11:46:25.000",
        "2020-11-12T12:04:04.000"
    ],
    "values": [
        [
            "202.01",
            "1992.0"
        ],
        [
            "0",
            "1"
        ],
        [
            "17.5",
            "30.0"
        ],
        [
            "40",
            "122"
        ]
    ]
}