Parsing sensor data to a JSON file through MQTT Broker

[Adeel]

Hi Dev team,

I am working on a nrf9160DK and transmitting my sensor data to a MQTT Client. I tested this through an application : MQTT.fx (Using mqtt.eclipse.org as open source broker) and I was able to get my data published.

Now, I need to parse my sensor data to a JSON file using the same broker : mqtt.eclipse.org on topic : my/publish/topic.

I have attached the JSON file where my data needs to be passed.

8867.config.rar

I would like to know the steps that would be required to parse data from my code to a JSON file.

I used the mqtt_simple client example in NCS v1.3.0 to publish the data outside. Can it be modified to parse the data to a JSON file.

Regards,

Adeel.

[JONATHAN LL]

Hi, 

my/publish/topic.

 I would advise to not use the default topics and topic structure in the final project as it may end up getting random test data from anyone. Add some extra label or rename something.


I have not used MQTT.FX so i cant really give propper feedback on that. Here is a link to a different solution that might be equally helpful using python. 

I would like to know the steps that would be required to parse data from my code to a JSON file.

 Do you mean on the nrf9160 or on the MQTT.FX side? Do you intend to store JSON files on the nrf9160 or are you takling about putting sensor data in to a JSON format?


A proposed solution based on initial understanding of the question:
- Take measurement
- Send as MQTT message to broker Use another client (such as MQTT.fx) to receive the message
- Take the necessary information from the MQTT payload
- Pass this information to a function that generates your JSON file

The last two steps i assume can be done in MQTT.fx


Take a look this example, it includes a more complex solution that uses JSON. There has been som changes in the newer version v1.4.0 that improves the example so might be smart to look at latest release tag. 

What is the end goal? Do you intend to use a desktop/phone application to view sensors?

Here is a link to a guide using AWS Cloud solution, it might be of interest. 

Regards,
Jonathan

[Adeel]

Hi Jonathon,

Thanks for the detailed reply . Gave me a lot of insight.

I was actually looking to write my sensor data to this JSON file through the nrf9160dk example.

config.rar

The JSON has a mqtt field : 

"mqtt": {
"host": "mqtt.eclipse.org",
"port": 1883,
"topic": "my/publish/topic"

through which I want to identify the broker to which I am sending the data.

After that, I just want to add my data by creating a value tag for a sensor.

So, as a first step, I wanted to know as to how to create this type of JSON file through C that I could through SES.

Also, how to write values into this kind of a file.

Regards,

Adeel.

[JONATHAN LL]

Hi Adeel,


It is not a file like in a file system file, it is a object witch you can store data in. You can follow the examples in aws_iot(link to use here) and asset_tracker (link to use in assetracker) to see how it is used to for with JSON in C. If you want your own file that contains JSON data and want to read and write to that file then you need to use a file system. 

Here is the link to the documentation for cJSON. I recommend reading this, the implementation used in the aws_iot and asset_tracker examples are a bit much to get into at first.

Here is a similar case on Devzone: Mqtt simple nrf9160, send JSON as utf-8 - Nordic Q&A - Nordic DevZone - Nordic DevZone (nordicsemi.com) 

If you want a file system then maybe look in to Zephyr RTOS Virtual Filesystem Switch (VFS) , but i dont hink that it is needed if i understand your use case correctly.

Regards,
Jonathan

[Adeel]

Hi Jonathon,

Okay, now I get your point. Yes, I think I don't need a file system. 

I will study the cJSON library and try to send my data out through it. 

1 question regarding my existing JSON file that I linked above:

It has an mqtt field : 

"mqtt": {
"host": "mqtt.eclipse.org",
"port": 1883,
"topic": "my/publish/topic"

Is this field enough for the board to recognise this broker and send the data to this file.

I know its a naive question, but I have never worked with file systems as such :). Just worked with inbuilt software's so was not sure if the writing happens in this way.

Regards,

Adeel.

[JONATHAN LL]

Hi Adeel,

Could you link to what example you are getting the config.json file from? 

Lets say you wat to use MQTT on the nrf9160, we can start by suing the mqtt_simple example provided in NCS as you said you are using. If we go tot he prj.conf file we can see a section called #application. All you have to do is fill out the information about the broker there( it is by default commented out). Then there is no need for a dedicated config.json file, everything can be done in the prj.conf file. 

# Appliaction
CONFIG_MQTT_PUB_TOPIC="/my/publish/topic"
CONFIG_MQTT_SUB_TOPIC="/my/subscribe/topic"
CONFIG_MQTT_CLIENT_ID="my-client-id"
CONFIG_MQTT_BROKER_HOSTNAME="mqtt.eclipse.org"
CONFIG_MQTT_BROKER_PORT=1883

MQTT.fx (Using mqtt.eclipse.org as open source broker) and I was able to get my data published.

 This refers to MQTT.fx working  and not nrf9160 mqtt_simple working ?


There are no stupid questions, feel free to ask anything

Regards,
Jonathan

[Adeel]

Hi Jonathon,

JONATHAN LL said:

Could you link to what example you are getting the config.json file from? 

I had a python code through which I was able to generate the file. It used the paho library to connect to the broker.

Fullscreen mqttgen.py Download

#!/usr/bin/env python3
"""a simple sensor data generator that sends to an MQTT broker via paho"""
import sys
import json
import time
import random
import time
import datetime

import paho.mqtt.client as mqtt

def generate(host, port, username, password, topic, sensors, interval_ms, verbose):
    """generate data and send it to an MQTT broker"""
    mqttc = mqtt.Client()

    if username:
        mqttc.username_pw_set(username, password)

    mqttc.connect(host, port)

    keys = list(sensors.keys())
    interval_secs = interval_ms / 1000.0

    while True:
        sensor_id = random.choice(keys)
        sensor = sensors[sensor_id]
        min_val, max_val = sensor.get("range", [0, 100])
        val = random.randint(min_val, max_val)
        ts=time.time()
        st = datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
        data = {
            "Time": st,
            "id": sensor_id,
            "value": val
        }

        for key in ["lat", "lng", "unit", "type", "description"]:
            value = sensor.get(key)

            if value is not None:
                data[key] = value

        payload = json.dumps(data)

        if verbose:
            print("%s: %s" % (topic, payload))

        mqttc.publish(topic, payload, qos=2, retain=True)
        mqttc.loop()
        time.sleep(interval_secs)


def main(config_path):
    """main entry point, load and validate config and call generate"""
    try:
        with open(config_path) as handle:
            config = json.load(handle)
            mqtt_config = config.get("mqtt", {})
            misc_config = config.get("misc", {})
            sensors = config.get("sensors")

            interval_ms = misc_config.get("interval_ms", 500)
            verbose = misc_config.get("verbose", False)

            if not sensors:
                print("no sensors specified in config, nothing to do")
                return

            host = mqtt_config.get("host", "localhost")
            port = mqtt_config.get("port", 1883)
            username = mqtt_config.get("username")
            password = mqtt_config.get("password")
            topic = mqtt_config.get("topic", "mqttgen")

            generate(host, port, username, password, topic, sensors, interval_ms, verbose)
    except IOError as error:
        print("Error opening config file '%s'" % config_path, error)

if __name__ == '__main__':
    if len(sys.argv) == 2:
        main(sys.argv[1])
    else:
        print("usage %s config.json" % sys.argv[0])

Yes, my prj.conf file has the same configuration. Actually, I had tested my data transfer through MQTT.fx with : mqtt.eclipse.org and that worked.

I wanted to transfer the same data from my nrf9160DK now directly to this JSON file and not go through MQTT.fx at all. 

I also needed to segregate different data in a JSON format like how it is in the attached JSON file and send out. So, I think cJSON library is a good starting point for this. 

JONATHAN LL said:

This refers to MQTT.fx working  and not nrf9160 mqtt_simple working ?

Yes, sending data to MQTT.fx is working with nrd9160 mqtt_simple.

Regards,

Adeel.

[JONATHAN LL]

Hi Addel,

There si no need send the broker info in JSON format. The MQTT example is setting up a MQTT client on the nrf9160 much like the MQTT.fx does on your computer. First the client tries to connect to the server(broker), and when it achieves a connection it will subscribe to a topic (my/publish/topic). Then there is no need to send the broker port and topic together with the message(sensor data). You can do it but since it the broker and topic info is all ready known it should be no use to send it as well. 

You can also test the mqtt_simple example with HIVE mqtt websocket. 

Here is a link to a extra cJSON Devzone case. 

Adeel said:

I also needed to segregate different data in a JSON format like how it is in the attached JSON file and send out. So, I think cJSON library is a good starting point for this

Yes, cJSON will work for this. 


Regards,
Jonathan

[Adeel]

Hi Jonathon,

Thanks for the information. now, I am clear with this. The broker and topics are already configured in the prj.conf, so my task is to just parse the data accordingly. 

I will try this and let you know, but real thanks for clearing up my queries regarding this . Really appreciate it.

Regards,

Adeel.

[Adeel]

Hi Jonathon,

Just to let you know that I got it working, thanks for the guidance on this .

Regards,

Adeel.