nRF Cloud custom MQTT topics

[Oskar]

Hey!

I am currently working on a device that will produce quite a large amount of data. We are talking 100Hz continuous sampling of 6 sensors + timestamps, so we have a custom data format to reduce the size a bit and we end up producing about 4-5 MB of data per hour which we want to transmit to the cloud. On top of this, the data must be encrypted.

My idea is to transmit the data to a custom MQTT topic in the cloud, something like ':device_id/:data_format_version/measurements'. A backend service will be subscribed to this topic and decrypt, decode and then store data in some persistent storage as soon as it comes in.

I am not sure whether or not this solution will work yet, from reading this page: https://docs.nrfcloud.com/APIs/MQTT/Topics/ it is not clear whether or not we can use a custom MQTT topic with nRF Cloud. So my question is whether or not it would be possible to do something like this in nRF Cloud. If custom topics are not possible, do you have any tips on how we can push a large amount of binary data in nRF Cloud?

[dejans]

Hi,

Device messages can be sent either by REST API or MQTT API. Please be aware of potential data loss that can happen in situations mentioned in sending device messages. If your intention is to store data in nRF Cloud, then you should use MQTT Message topics. 

Best regards,
Dejan

[Oskar]

So I can use any custom topic with the nRF Cloud and it is considered a device message?

I don't think this is quite clear from the documentation as it states: Topics that begin with $aws are AWS reserved topics. All other topics are custom-defined by nRF Cloud.

From how I interpret your response I can set up my devices to transmit to, for example, 04267166-5df8-541d-8dd9-44492dd4af9d/2/measurements and that would work fine, with the possibility of data loss as mentioned in the documentation.

Is this correct?

And on the topic of data loss, I assume the data is only lost from storage and all messages should be received by any subscriber to the MQTT topic(given a correctly configured MQTT QoS)?

[dejans]

Hi,

The amount of data you would want to send to nRF Cloud is quite big, and it is currently not supported.

Messages sent to any topic would need to be prefixed with ${mqttTopicPrefix}/m. Therefore, you would need to adjust your topic to ${mqttTopicPrefix}/m/:device_id/:data_format_version/measurements, and probably send data in chunks less than about 350 KB.

There is currently no official support for binary data, and there are no guarantees how this would work in your case. If you want to go this way, you would need to experiment. 

Best regards,
Dejan

[Oskar]

Okay, I see!

Thank you for your time, I will do some experiments and see how it goes.