Parsing sensor data to a JSON file through MQTT Broker

[Adeel]

Hi Dev team,

I am working on a nrf9160DK and transmitting my sensor data to a MQTT Client. I tested this through an application : MQTT.fx (Using mqtt.eclipse.org as open source broker) and I was able to get my data published.

Now, I need to parse my sensor data to a JSON file using the same broker : mqtt.eclipse.org on topic : my/publish/topic.

I have attached the JSON file where my data needs to be passed.

8867.config.rar

I would like to know the steps that would be required to parse data from my code to a JSON file.

I used the mqtt_simple client example in NCS v1.3.0 to publish the data outside. Can it be modified to parse the data to a JSON file.

Regards,

Adeel.

[JONATHAN LL]

Hi Adeel,

Could you link to what example you are getting the config.json file from? 

Lets say you wat to use MQTT on the nrf9160, we can start by suing the mqtt_simple example provided in NCS as you said you are using. If we go tot he prj.conf file we can see a section called #application. All you have to do is fill out the information about the broker there( it is by default commented out). Then there is no need for a dedicated config.json file, everything can be done in the prj.conf file. 

# Appliaction
CONFIG_MQTT_PUB_TOPIC="/my/publish/topic"
CONFIG_MQTT_SUB_TOPIC="/my/subscribe/topic"
CONFIG_MQTT_CLIENT_ID="my-client-id"
CONFIG_MQTT_BROKER_HOSTNAME="mqtt.eclipse.org"
CONFIG_MQTT_BROKER_PORT=1883

MQTT.fx (Using mqtt.eclipse.org as open source broker) and I was able to get my data published.

 This refers to MQTT.fx working  and not nrf9160 mqtt_simple working ?


There are no stupid questions, feel free to ask anything

Regards,
Jonathan

[Adeel]

Hi Jonathon,

JONATHAN LL said:

Could you link to what example you are getting the config.json file from? 

I had a python code through which I was able to generate the file. It used the paho library to connect to the broker.

Fullscreen mqttgen.py Download

#!/usr/bin/env python3
"""a simple sensor data generator that sends to an MQTT broker via paho"""
import sys
import json
import time
import random
import time
import datetime

import paho.mqtt.client as mqtt

def generate(host, port, username, password, topic, sensors, interval_ms, verbose):
    """generate data and send it to an MQTT broker"""
    mqttc = mqtt.Client()

    if username:
        mqttc.username_pw_set(username, password)

    mqttc.connect(host, port)

    keys = list(sensors.keys())
    interval_secs = interval_ms / 1000.0

    while True:
        sensor_id = random.choice(keys)
        sensor = sensors[sensor_id]
        min_val, max_val = sensor.get("range", [0, 100])
        val = random.randint(min_val, max_val)
        ts=time.time()
        st = datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
        data = {
            "Time": st,
            "id": sensor_id,
            "value": val
        }

        for key in ["lat", "lng", "unit", "type", "description"]:
            value = sensor.get(key)

            if value is not None:
                data[key] = value

        payload = json.dumps(data)

        if verbose:
            print("%s: %s" % (topic, payload))

        mqttc.publish(topic, payload, qos=2, retain=True)
        mqttc.loop()
        time.sleep(interval_secs)


def main(config_path):
    """main entry point, load and validate config and call generate"""
    try:
        with open(config_path) as handle:
            config = json.load(handle)
            mqtt_config = config.get("mqtt", {})
            misc_config = config.get("misc", {})
            sensors = config.get("sensors")

            interval_ms = misc_config.get("interval_ms", 500)
            verbose = misc_config.get("verbose", False)

            if not sensors:
                print("no sensors specified in config, nothing to do")
                return

            host = mqtt_config.get("host", "localhost")
            port = mqtt_config.get("port", 1883)
            username = mqtt_config.get("username")
            password = mqtt_config.get("password")
            topic = mqtt_config.get("topic", "mqttgen")

            generate(host, port, username, password, topic, sensors, interval_ms, verbose)
    except IOError as error:
        print("Error opening config file '%s'" % config_path, error)

if __name__ == '__main__':
    if len(sys.argv) == 2:
        main(sys.argv[1])
    else:
        print("usage %s config.json" % sys.argv[0])

Yes, my prj.conf file has the same configuration. Actually, I had tested my data transfer through MQTT.fx with : mqtt.eclipse.org and that worked.

I wanted to transfer the same data from my nrf9160DK now directly to this JSON file and not go through MQTT.fx at all. 

I also needed to segregate different data in a JSON format like how it is in the attached JSON file and send out. So, I think cJSON library is a good starting point for this. 

JONATHAN LL said:

This refers to MQTT.fx working  and not nrf9160 mqtt_simple working ?

Yes, sending data to MQTT.fx is working with nrd9160 mqtt_simple.

Regards,

Adeel.

[JONATHAN LL]

Hi Addel,

There si no need send the broker info in JSON format. The MQTT example is setting up a MQTT client on the nrf9160 much like the MQTT.fx does on your computer. First the client tries to connect to the server(broker), and when it achieves a connection it will subscribe to a topic (my/publish/topic). Then there is no need to send the broker port and topic together with the message(sensor data). You can do it but since it the broker and topic info is all ready known it should be no use to send it as well. 

You can also test the mqtt_simple example with HIVE mqtt websocket. 

Here is a link to a extra cJSON Devzone case. 

Adeel said:

I also needed to segregate different data in a JSON format like how it is in the attached JSON file and send out. So, I think cJSON library is a good starting point for this

Yes, cJSON will work for this. 


Regards,
Jonathan

[Adeel]

Hi Jonathon,

Thanks for the information. now, I am clear with this. The broker and topics are already configured in the prj.conf, so my task is to just parse the data accordingly. 

I will try this and let you know, but real thanks for clearing up my queries regarding this . Really appreciate it.

Regards,

Adeel.

[Adeel]

Hi Jonathon,

Just to let you know that I got it working, thanks for the guidance on this .

Regards,

Adeel.