The four kinds of data you need to consider when developing an IoT product

In this post I will give you an introduction to the four kinds of data you need to consider when developing an IoT product:

1. Device State
2. Device Configuration
3. Past Data
4. Firmware Updates

It's these messages that are exchanged between your devices and your backend which are the most important aspect to optimize for when developing an ultra-low-power product because initiating and maintaining network connection is relatively expensive compared to other device operations (for example reading a sensor value). It is therefore recommended to invest a reasonable amount of time to revisit the principles explained here and customize them to your specific needs. The more the modem-uptime can be reduced and the smaller the total transferred amount if data becomes, the longer will your battery and your data contingent last.

The four kinds of data

1. Device State

A device needs to send its sensor readings (like position, temperature) and information about it's health to the backend, first an foremost is the battery level a critical health indicator. This data is considered the device state. Because we want to always be able to quickly see the latest state of the device, a digital twin can be used to store this state on the backend side: whenever the device sends an update, the digital twin is updated. This allows an application to access the most recent device state immediately without needing to wait for the device to connect and publish its state.

It is an important criterion for the robustness of any IoT product to gracefully handle situations in which the device is not connected to the internet. It might even not be favorable to be connected all the time—wireless communication is relatively expensive consumes a lot of energy and therefore increases the power consumption.

To optimize for ultra low power consumption, we want to turn off the modem as quickly as possible and keep it off as long as possible.

This can be achieved by making the device smart and allowing it to decide based on the situation whether it should try to send data.

For example could an asset tracker use the motion sensor to decide whether to publish its state frequently or if it detects no movement for a while go into a passive mode, where it turns of the modem and waits until it detects movement again. It could also use the internal clock to wake up every hour to sent a heartbeat, after all we might want to know that the device is healthy, even it is not in motion.

2. Device Configuration

Depending on the product we might also want to change the device configuration. This could on the one hand be use during development to tweak the aforementioned behavior using variables instead of pushing a new firmware over the air to the device. We observe firmware sizes of around 500 KB which will, even when compressed, be expensive because it will take a device some time to download and apply the updated, not to mention the costs for transferring the firmware update over the cellular network. Especially in NB-IoT-only deployments is the data rate low. Updating a fleet of devices with a new firmware involves orchestrating the roll-out and observing for faults. All these challenges lead to the need to be able to configure the device, which allows to tweak the behavior of the device until the inflection point is reached: battery life vs. data granularity. Interesting configuration options are for example the sensitivity of the motion sensor: depending on the tracked subject what is considered "movement" can vary greatly. Various timeout settings have an important influence on power- and data-consumption: the time the device waits to acquire a GPS fix, or the time it waits between sending updates when in motion.

On the other hand is device configuration needed if the device controls something: imaging a smart lock which needs to manipulate the state of a physical lock. The backend needs a way to tell the device which state that lock should be in, and this setting needs to be persisted on the backend, since the device could lose power, crash or otherwise lose the information if the lock should be open or closed.

Here again is the digital twin used on the cloud side to store the latest desired configuration of the device immediately, so the application does not have to wait for the device to be connected to record the configuration change. The implementation of the digital twin then will take care of sending only the latest required changes to the device (all changes since the device did last request its configuration are combined into one change) thus also minimizing the amount of data which needs to be transferred to the device.

Timestamping

Device state and configuration are timeless datum, they apply always and absolutely. The device sends a GPS position over the cellular connection and the digital twin is updated, we now know where the device is now. When the device configuration is changed (A -> A') the device will eventually apply the new configuration, and if another configuration change was made while the device was not connected (A' -> A'') the device can directly jump to A''. To make state and configuration changes available over time we can store all changes on the backend with the time of the change and make them available for retrieval in a time-series fashion.

Time is relative.

This approach has an inherent problem: if we are to store the battery level measured by the device with the time it was received by the cloud, the timestamp will not be accurate. It can take minutes between the sampling of the battery voltage and the time the update is finally delivered on the receiving end, because for example it took the device a while to establish the cellular connection in order to send the update. While this might be acceptable with a sensor that has low volatility, it might not be acceptable in scenarios where it is important to exactly know when something happened. Imaging you are tracking parcels and want to track if a parcel is dropped. A few minutes can make a big difference to pinpoint the exact situation when the parcel is being moved by a person or even in a vehicle.

The need for precise time measurement on the device is important and is achieved by combining three time sources: the relative device timestamp (a relative time with microsecond resolution that counts upwards from zero after the device is powered on), the cellular network time and the time from the GPS sensor.

       42      131
0---------------------->
        ^       ^
        |       |
       LTE     GPS

Every time a sensor is read, the value is recorded with the device timestamp. Once theses measurements are about to be sent (in which case there is a cellular connection and at least the network time is known), the relative timestamps can be converted to absolute timestamps using the relative timestamps of the network or the GPS time.

This way all data is sent with precise timestamps to the cloud where the device time is used when visualizing data to accurately reflect when the datum was created.

3. Past State

Imagine a reindeer tracker which tracks the position of a herd. If position updates are only collected when a cellular connection can be established there will be an interesting observation: the reindeers are only walking along ridges, but never in valleys. The reason is not because they don't like the valley, but because the cellular signal does not reach deep down into remote valleys. The GPS signal however will be received there from the tracker because satellites are high on the horizon and can send their signal down into the valley.

There are many scenarios where cellular connection might not be available or unreliable but reading sensors work. Robust ultra-mobile IoT products therefore must make this a normal mode of operation: the absence of a cellular connection must be treated as a temporary condition which will eventually resolve and until then business as usual ensues. This means devices should keep measuring and storing these measures in a ring-buffer or employ other strategies to decide which data to discard once the memory limit is reached.

Once the device is successfully able to establish a connection it will then (after publishing its most recent measurements) publish past data in batch.

On a side note: the same is true for devices that control a system. They should have built-in decision rules and must not depend on an answer from a cloud backend to provide the action to execute based on the current condition.

4. Firmware Updates

Arguably a firmware update over the air can be seen as configuration, however the size of a typical firmware image (500KB) is 2-3 magnitudes larger than a control message. Therefore it can be beneficial to treat it differently. Typically an update is initiated by a configuration change, once acknowledged by the device will initiate the firmware download. The download itself is done out of band using not MQTT but HTTP(s) to reduce overhead.

Additionally firmware updates are so large compared to other messages that the device may suspend all other operation until the firmware update has been applied to conserve resources.