Multi instance backend subscribing to MQTT topic

RE: Multi instance backend subscribing to MQTT topic

Lincoln Thomas — Fri, 16 Aug 2024 01:59:46 GMT

Hi Rasmus,

I have created 3 more account devices and sent their credentials to your email.

If they work for you, and you still need more account devices, I can create them, but it is a manual process.

We will soon be supporting user creation of multiple account devices.

Regards,

Lincoln

RE: Multi instance backend subscribing to MQTT topic

Charlie — Fri, 09 Aug 2024 07:08:58 GMT

Hi Rasmus,

Here is a reply from our nRF Cloud developer, could you provide your team ID and account devices amount you what you need?

Only 1 account device is allowed now, and the supported REST API endpoints to use are:

GetAccountDeviceCertificate
RotateAccountDeviceCertificate
DeleteAccountDevice

The mentions in the DevZone ticket about CreateAccountDeviceAndCertificate and related endpoints are deprecated equivalents that should not be used. They do NOT allow creation of more than one account device, just like the 3 bulleted supported endpoints above do not allow more than one.

We are considering adding feature support to allow creation of multiple account devices, but it's not possible now by the user. But we can create the 2nd, 3rd, etc account devices for them manually and give them the client device IDs and certs, if we have their team ID.

Best regards,

Charlie

RE: Multi instance backend subscribing to MQTT topic

Charlie — Tue, 06 Aug 2024 14:00:34 GMT

Hi Rasmus,

I will forward your request to our nRF Cloud team to see if we could figure out a solution.

Best regards,

Charlie

RE: Multi instance backend subscribing to MQTT topic

Rasmus K — Sat, 03 Aug 2024 11:29:46 GMT

Hi Charlie, that's okay.

Yes I saw that endpoint, but it says it's deprecated and should not be used. The suggested endpoint to use instead https://api.nrfcloud.com/v1#tag/Account-Devices/operation/CreateAccountDevice - specifically says:

"When a new account device (and subsequent certificate) is created, both the public AND private keys are returned. If an account device already exists, only the public key is returned. If the private key is lost, the certificate should be rotated to obtain a new public/private key pair".

This conforms with the result of my attempt trying it, i.e. the second time I call the endpoint I get the same clientId back as well as the public key. Limiting me to only have one of these account devices created. So, there is no way for me to connect to the NRF MQTT broker with different device ID:s (client ID:s) since those are tied to a account device, which I can only create one of.

Is this by design, or?

Best regards,

Rasmus

EDIT: I did try the deprecated endpoint as well that you provided, that does not create a new account device either if I already have one. It only returns the same information of the one already created.

RE: Multi instance backend subscribing to MQTT topic

Charlie — Fri, 02 Aug 2024 13:45:39 GMT

Hi Rasmus,

Sorry for misunderstanding. I forget you are using MQTT clients on your cloud side.

For you questions,

[quote user="Rasmus K"]Is there no way to create more account devices?[/quote]

There is no limitation to create account devices on nRF Cloud nRF Cloud REST API Documentation, but you are trying to run MQTT clients with same account device ID, this is not allowed by MQTT Spec.

How to handle multiple clients with same ID in mosquitto MQTT? - Stack Overflow

[quote user="Rasmus K"]Is there another, recommended way for my backend to subscribe to live location events instead of subscribing to appropriate MQTT topics?[/quote]

If you are sensitive to the location update time delay, a direct connection and update to your cloud could be the solution. Even the MQTT PUB/SUB introduces delay.

Best regards,

Charlie

RE: Multi instance backend subscribing to MQTT topic

Rasmus K — Fri, 02 Aug 2024 10:06:42 GMT

Yes, we have thought about self hosting our own MQTT broker. However we wanted to explore a managed solution first. FOTA etc is supported out of the box for most managed solutions AFAIK.

Is it possible to provide answers to my questions? AWS IoT (which NRF cloud is based on AFAIK) does not have a limitation on how many "things" can be created (which I guess is the account device in NRF cloud). So another solution would be to use our own AWS IoT infra, bypassing NRF cloud, right?

RE: Multi instance backend subscribing to MQTT topic

Charlie — Thu, 01 Aug 2024 12:13:34 GMT

Have you thought about using EMQX which already supports K8s deployment?

Running MQTT on Kubernetes: A Practical Guide | EMQ (emqx.com)
EMQX is the leading open source MQTT broker. It provides a Docker Official Image which is available on Docker Hub and an open source Kubernetes operator, which supports the free and open source version of EMQX.

RE: Multi instance backend subscribing to MQTT topic

Rasmus K — Thu, 01 Aug 2024 10:25:12 GMT

Hi again Charlie,

I revised my solution to enable all my replicas to subscribe to MQTT topics of devices which are connected to my API and should be provided with live location updates. However I have encountered an issue, namely that I can only create one account device. As the MQTT broker prevents me from connecting with the same client ID, I need to use different ones (and different account devices, right?) so my thought was to have multiple, pooled account devices which my replicas can pick from when connecting to the MQTT broker. Which leads me to some questions:

Is there no way to create more account devices?
Is there another, recommended way for my backend to subscribe to live location events instead of subscribing to appropriate MQTT topics?

Thanks in advance,

Rasmus

RE: Multi instance backend subscribing to MQTT topic

Rasmus K — Sat, 27 Jul 2024 16:02:13 GMT

Yes, for now I am using a simple leader election mechanism (backed by postgresql) - which means one of my replicas will be elected as a leader, and the leader subscribes to the MQTT topics. In the future I might want to use compute across my replicas to receive MQTT messages, then I need to change this approach.

Best regards,

Rasmus

RE: Multi instance backend subscribing to MQTT topic

Charlie — Thu, 25 Jul 2024 07:06:52 GMT

Hi Rasmus,

We can keep this ticket open to see if there are any other developers will share their solutions.

Feel free to give an update if you solve this challenge by yourself.

Best regards,

Charlie

RE: Multi instance backend subscribing to MQTT topic

Rasmus K — Wed, 24 Jul 2024 15:21:39 GMT

Hi Charlie,

Thanks I understand. I was thinking there must be others wanting to subscribe to the MQTT topics from device accounts to process the data in their applications. And nowadays it's just so common to run multiple replicas/instances of your application.

Do you think we should keep this open for a while or close it?

Best regards,

Rasmus

RE: Multi instance backend subscribing to MQTT topic

Charlie — Wed, 24 Jul 2024 13:11:42 GMT

Hi Rasmus,

We do not have much experience on multiple instances MQTT client implementation since our focus is on single MQTT client on IoT device mostly.

You may reach to MQTT or Kubernetes forum to get more professional advice.

Best regards,
Charlie