Parsing sensor data to a JSON file through MQTT Broker

RE: Parsing sensor data to a JSON file through MQTT Broker

Adeel — Thu, 26 Nov 2020 15:28:56 GMT

Hi Jonathon,

Just to let you know that I got it working, thanks for the guidance on this 🙂.

Regards,

Adeel.

RE: Parsing sensor data to a JSON file through MQTT Broker

Adeel — Thu, 26 Nov 2020 10:43:51 GMT

Hi Jonathon,

Thanks for the information. now, I am clear with this. The broker and topics are already configured in the prj.conf, so my task is to just parse the data accordingly.

I will try this and let you know, but real thanks for clearing up my queries regarding this 🙂. Really appreciate it.

Regards,

Adeel.

RE: Parsing sensor data to a JSON file through MQTT Broker

JONATHAN LL — Thu, 26 Nov 2020 08:22:56 GMT

Hi Addel,

There si no need send the broker info in JSON format. The MQTT example is setting up a MQTT client on the nrf9160 much like the MQTT.fx does on your computer. First the client tries to connect to the server(broker), and when it achieves a connection it will subscribe to a topic (my/publish/topic). Then there is no need to send the broker port and topic together with the message(sensor data). You can do it but since it the broker and topic info is all ready known it should be no use to send it as well.

You can also test the mqtt_simple example with HIVE mqtt websocket.

Here is a link to a extra cJSON Devzone case.

[quote user="Adeel"]I also needed to segregate different data in a JSON format like how it is in the attached JSON file and send out. So, I think cJSON library is a good starting point for this[/quote]

Yes, cJSON will work for this.

Regards,
Jonathan

RE: Parsing sensor data to a JSON file through MQTT Broker

Adeel — Wed, 25 Nov 2020 15:32:36 GMT

Hi Jonathon,

[quote userid="91187" url="~/f/nordic-q-a/68689/parsing-sensor-data-to-a-json-file-through-mqtt-broker/281874#281874"]Could you link to what example you are getting the config.json file from? [/quote]

I had a python code through which I was able to generate the file. It used the paho library to connect to the broker.

devzone.nordicsemi.com/.../mqttgen.py

Yes, my prj.conf file has the same configuration. Actually, I had tested my data transfer through MQTT.fx with : mqtt.eclipse.org and that worked.

I wanted to transfer the same data from my nrf9160DK now directly to this JSON file and not go through MQTT.fx at all.

I also needed to segregate different data in a JSON format like how it is in the attached JSON file and send out. So, I think cJSON library is a good starting point for this.

[quote userid="91187" url="~/f/nordic-q-a/68689/parsing-sensor-data-to-a-json-file-through-mqtt-broker/281874#281874"]This refers to MQTT.fx working and not nrf9160 mqtt_simple working ?
[/quote]

Yes, sending data to MQTT.fx is working with nrd9160 mqtt_simple.

Regards,

Adeel.

RE: Parsing sensor data to a JSON file through MQTT Broker

JONATHAN LL — Wed, 25 Nov 2020 15:08:18 GMT

Hi Adeel,

Could you link to what example you are getting the config.json file from?

Lets say you wat to use MQTT on the nrf9160, we can start by suing the mqtt_simple example provided in NCS as you said you are using. If we go tot he prj.conf file we can see a section called #application. All you have to do is fill out the information about the broker there( it is by default commented out). Then there is no need for a dedicated config.json file, everything can be done in the prj.conf file.

# Appliaction
CONFIG_MQTT_PUB_TOPIC="/my/publish/topic"
CONFIG_MQTT_SUB_TOPIC="/my/subscribe/topic"
CONFIG_MQTT_CLIENT_ID="my-client-id"
CONFIG_MQTT_BROKER_HOSTNAME="mqtt.eclipse.org"
CONFIG_MQTT_BROKER_PORT=1883

[quote user=""] MQTT.fx (Using mqtt.eclipse.org as open source broker) and I was able to get my data published.[/quote]

This refers to MQTT.fx working and not nrf9160 mqtt_simple working ?

There are no stupid questions, feel free to ask anything

Regards,
Jonathan

RE: Parsing sensor data to a JSON file through MQTT Broker

Adeel — Wed, 25 Nov 2020 14:45:23 GMT

Hi Jonathon,

Okay, now I get your point. Yes, I think I don't need a file system.

I will study the cJSON library and try to send my data out through it.

1 question regarding my existing JSON file that I linked above:

It has an mqtt field :

"mqtt": {
"host": "mqtt.eclipse.org",
"port": 1883,
"topic": "my/publish/topic"

Is this field enough for the board to recognise this broker and send the data to this file.

I know its a naive question, but I have never worked with file systems as such :). Just worked with inbuilt software's so was not sure if the writing happens in this way.

Regards,

Adeel.

RE: Parsing sensor data to a JSON file through MQTT Broker

JONATHAN LL — Wed, 25 Nov 2020 12:25:52 GMT

Hi Adeel,

It is not a file like in a file system file, it is a object witch you can store data in. You can follow the examples in aws_iot(link to use here) and asset_tracker (link to use in assetracker) to see how it is used to for with JSON in C. If you want your own file that contains JSON data and want to read and write to that file then you need to use a file system.

Here is the link to the documentation for cJSON. I recommend reading this, the implementation used in the aws_iot and asset_tracker examples are a bit much to get into at first.

Here is a similar case on Devzone: Mqtt simple nrf9160, send JSON as utf-8 - Nordic Q&A - Nordic DevZone - Nordic DevZone (nordicsemi.com)

If you want a file system then maybe look in to Zephyr RTOS Virtual Filesystem Switch (VFS) , but i dont hink that it is needed if i understand your use case correctly.

Regards,
Jonathan

RE: Parsing sensor data to a JSON file through MQTT Broker

Adeel — Wed, 25 Nov 2020 11:50:03 GMT

Hi Jonathon,

I was progressing with this and I found a cjson library in the nrf folder : C:\Users\adeel\ncs\v1.3.0\nrf\ext\cjson

It contains the cJSON.c and cJSON.h file.

Can this library file be used to generate a JSON file of its own ? Then I can create a file and add my fields in the same file.

Regards,

Adeel.

RE: Parsing sensor data to a JSON file through MQTT Broker

Adeel — Tue, 24 Nov 2020 23:26:23 GMT

Hi Jonathon,

Thanks for the detailed reply 🙂. Gave me a lot of insight.

I was actually looking to write my sensor data to this JSON file through the nrf9160dk example.

devzone.nordicsemi.com/.../config.rar

The JSON has a mqtt field :

"mqtt": {
"host": "mqtt.eclipse.org",
"port": 1883,
"topic": "my/publish/topic"

through which I want to identify the broker to which I am sending the data.

After that, I just want to add my data by creating a value tag for a sensor.

So, as a first step, I wanted to know as to how to create this type of JSON file through C that I could through SES.

Also, how to write values into this kind of a file.

Regards,

Adeel.

RE: Parsing sensor data to a JSON file through MQTT Broker

JONATHAN LL — Tue, 24 Nov 2020 20:54:00 GMT

Hi,

[quote user=""]my/publish/topic.[/quote]

I would advise to not use the default topics and topic structure in the final project as it may end up getting random test data from anyone. Add some extra label or rename something.

I have not used MQTT.FX so i cant really give propper feedback on that. Here is a link to a different solution that might be equally helpful using python.

[quote user=""]I would like to know the steps that would be required to parse data from my code to a JSON file.[/quote]

Do you mean on the nrf9160 or on the MQTT.FX side? Do you intend to store JSON files on the nrf9160 or are you takling about putting sensor data in to a JSON format?

A proposed solution based on initial understanding of the question:
- Take measurement
- Send as MQTT message to broker Use another client (such as MQTT.fx) to receive the message
- Take the necessary information from the MQTT payload
- Pass this information to a function that generates your JSON file

The last two steps i assume can be done in MQTT.fx

Take a look this example, it includes a more complex solution that uses JSON. There has been som changes in the newer version v1.4.0 that improves the example so might be smart to look at latest release tag.

What is the end goal? Do you intend to use a desktop/phone application to view sensors?

Here is a link to a guide using AWS Cloud solution, it might be of interest.

Regards,
Jonathan