Multi instance backend subscribing to MQTT topic

[Rasmus K]

Hi, I am wondering about the best way to handle the case where my backend is subscribing to MQTT topics in NFR cloud, when I am running multiple instances of it, for example in Kubernetes.

The requirement is that one of the instances should get one message, not all - in a load balanced fashion.

I have read about shared subscriptions and tried to use it. Seems to be working, however I am facing an issue where all of my instances keeps disconnecting and reconnecting. I suspect this is because the MQTT broker only allows one connection per client id, and I am using the same for all my instances (btw I am using a device account to subscribe to the topics from my backend).

My initial thoughts for a solution is:

1. Some form of leader election mechanism, to give one of my instances a "leader" role, hence subscribing to the topics and the other instances are not (works but complicated)
2. Generating as many device accounts as I plan to have replicas and use different device account per replica (I don't particularly like this because I don't want to create another account when scaling up, and the logic for grabbing an account feels non trivial)

Any other thoughts for a solution, that might be more straight forward?

Best regards,

Rasmus

[Charlie]

Hi Rasmus,

We do not have much experience on multiple instances MQTT client implementation since our focus is on single MQTT client on IoT device mostly.

You may reach to MQTT or Kubernetes forum to get more professional advice.

Best regards,
Charlie

[Rasmus K]

Hi Charlie,

Thanks I understand. I was thinking there must be others wanting to subscribe to the MQTT topics from device accounts to process the data in their applications. And nowadays it's just so common to run multiple replicas/instances of your application.

Do you think we should keep this open for a while or close it?

Best regards,

Rasmus

[Charlie]

Hi Rasmus,

We can keep this ticket open to see if there are any other developers will share their solutions.

Feel free to give an update if you solve this challenge by yourself.

Best regards,

Charlie

[Rasmus K]

Yes, for now I am using a simple leader election mechanism (backed by postgresql) - which means one of my replicas will be elected as a leader, and the leader subscribes to the MQTT topics. In the future I might want to use compute across my replicas to receive MQTT messages, then I need to change this approach.

Best regards,

Rasmus

[Rasmus K]

Hi again Charlie,

I revised my solution to enable all my replicas to subscribe to MQTT topics of devices which are connected to my API and should be provided with live location updates. However I have encountered an issue, namely that I can only create one account device. As the MQTT broker prevents me from connecting with the same client ID, I need to use different ones (and different account devices, right?) so my thought was to have multiple, pooled account devices which my replicas can pick from when connecting to the MQTT broker. Which leads me to some questions:

1. Is there no way to create more account devices?
2. Is there another, recommended way for my backend to subscribe to live location events instead of subscribing to appropriate MQTT topics?

Thanks in advance,

Rasmus

[Charlie]

Have you thought about using EMQX which already supports K8s deployment?

Running MQTT on Kubernetes: A Practical Guide | EMQ (emqx.com)
EMQX is the leading open source MQTT broker. It provides a Docker Official Image which is available on Docker Hub and an open source Kubernetes operator, which supports the free and open source version of EMQX.

[Rasmus K]

Yes, we have thought about self hosting our own MQTT broker. However we wanted to explore a managed solution first. FOTA etc is supported out of the box for most managed solutions AFAIK.

Is it possible to provide answers to my questions? AWS IoT (which NRF cloud is based on AFAIK) does not have a limitation on how many "things" can be created (which I guess is the account device in NRF cloud). So another solution would be to use our own AWS IoT infra, bypassing NRF cloud, right?

[Charlie]

Hi Rasmus,

Sorry for misunderstanding. I forget you are using MQTT clients on your cloud side.

For you questions,

Rasmus K said:

Is there no way to create more account devices?

There is no limitation to create account devices on nRF Cloud nRF Cloud REST API Documentation, but you are trying to run MQTT clients with same account device ID, this is not allowed by MQTT Spec.

How to handle multiple clients with same ID in mosquitto MQTT? - Stack Overflow

Rasmus K said:

Is there another, recommended way for my backend to subscribe to live location events instead of subscribing to appropriate MQTT topics?

If you are sensitive to the location update time delay, a direct connection and update to your cloud could be the solution. Even the MQTT PUB/SUB introduces delay.

Best regards,

Charlie

[Rasmus K]

Hi Charlie, that's okay.

Yes I saw that endpoint, but it says it's deprecated and should not be used. The suggested endpoint to use instead https://api.nrfcloud.com/v1#tag/Account-Devices/operation/CreateAccountDevice - specifically says:

"When a new account device (and subsequent certificate) is created, both the public AND private keys are returned. If an account device already exists, only the public key is returned. If the private key is lost, the certificate should be rotated to obtain a new public/private key pair".

This conforms with the result of my attempt trying it, i.e. the second time I call the endpoint I get the same clientId back as well as the public key. Limiting me to only have one of these account devices created. So, there is no way for me to connect to the NRF MQTT broker with different device ID:s (client ID:s) since those are tied to a account device, which I can only create one of.

Is this by design, or?

Best regards,

Rasmus

EDIT: I did try the deprecated endpoint as well that you provided, that does not create a new account device either if I already have one. It only returns the same information of the one already created.