Device intermittent failures to connect to nRF Cloud MQTT broker (err: -111)

[teba99]

Hi,

Intermittently, after a reset/restart some of our devices are unable to connect to the nRF Cloud MQTT broker:

[00:00:09.235,626] <inf> app_cloud_connection: Network connectivity gained!
[00:00:10.236,114] <inf> app_cloud_connection: Network is ready
[00:00:10.236,145] <inf> app_cloud_connection: Connecting to nRF Cloud
[00:01:00.567,901] <wrn> app_application: Cloud not ready within 60 seconds, proceeding with app startup
[00:01:10.252,746] <err> nrf_cloud_transport: Could not connect to nRF Cloud MQTT Broker mqtt.nrfcloud.com, port: 45858. err: -111
[00:01:10.252,807] <inf> app_cloud_connection: Disconnecting from nRF Cloud
[00:01:10.257,385] <inf> app_cloud_connection: Could not connect to nRF Cloud

[00:01:10.414,062] <inf> app_cloud_connection: Retrying nRF Cloud connection in 30 seconds...

Notes:

• This affects multiple devices.
• The error is intermittent. The devices connected fine just before the restart
• Once the error occurs, the device retries to connect but keeps getting this error indefinitely
• The device connection (LTE-M) appears to be working correctly as can be seen in the log, and we also see data flow in the SIM provider portal
• After another restart, the devices connect fine again.

Once a device is connected, we typically don't see this error.

It feels like there is a temporary network routing, firewall, or broker issue that prevents the device from connecting once. But the device network stack then gets stuck in a state where it now fails to reconnect again, even if the temporary problem went away. Restarting the device clears that state and allows the device to reconnect instantly.

Have you seen this before? Is there a workaround other then restarting the device?

Thanks

[teba99]

Following up on this: I now have a log that shows the device tries to connect for 10+ minutes and keeps getting err: -111. But suddenly -- without any changes on our side -- the error is gone and the device is able to connect:

[00:01:10.335,998] <inf> poc_app: Connecting over LTE..
[00:01:17.039,184] <inf> poc_app: LTE connection established. Connecting to nRF Cloud..
[00:02:11.483,795] <err> nrf_cloud_transport: Could not connect to nRF Cloud MQTT Broker mqtt.nrfcloud.com, port: 45858. err: -111
[00:03:17.039,306] <wrn> cloud_connection: Could not connect to nRF Cloud, readiness timeout
[00:03:17.039,337] <wrn> poc_app: Failed to establish connection to nRF Cloud
[00:03:22.039,428] <inf> poc_app: Connecting over LTE..
[00:03:22.048,675] <inf> poc_app: LTE connection established. Connecting to nRF Cloud..
[00:04:16.512,054] <err> nrf_cloud_transport: Could not connect to nRF Cloud MQTT Broker mqtt.nrfcloud.com, port: 45858. err: -111
[00:05:22.048,767] <wrn> cloud_connection: Could not connect to nRF Cloud, readiness timeout
[00:05:22.048,797] <wrn> poc_app: Failed to establish connection to nRF Cloud
[00:05:27.048,858] <inf> poc_app: Connecting over LTE..
[00:05:27.058,135] <inf> poc_app: LTE connection established. Connecting to nRF Cloud..
[00:06:21.523,834] <err> nrf_cloud_transport: Could not connect to nRF Cloud MQTT Broker mqtt.nrfcloud.com, port: 45858. err: -111
[00:07:27.058,227] <wrn> cloud_connection: Could not connect to nRF Cloud, readiness timeout
[00:07:27.058,258] <wrn> poc_app: Failed to establish connection to nRF Cloud
[00:07:32.058,349] <inf> poc_app: Connecting over LTE..
[00:07:32.067,596] <inf> poc_app: LTE connection established. Connecting to nRF Cloud..
[00:08:26.502,868] <err> nrf_cloud_transport: Could not connect to nRF Cloud MQTT Broker mqtt.nrfcloud.com, port: 45858. err: -111
[00:09:32.067,687] <wrn> cloud_connection: Could not connect to nRF Cloud, readiness timeout
[00:09:32.067,718] <wrn> poc_app: Failed to establish connection to nRF Cloud
[00:09:37.067,810] <inf> poc_app: Connecting over LTE..
[00:09:37.077,087] <inf> poc_app: LTE connection established. Connecting to nRF Cloud..
[00:10:31.627,929] <err> nrf_cloud_transport: Could not connect to nRF Cloud MQTT Broker mqtt.nrfcloud.com, port: 45858. err: -111
[00:11:37.077,178] <wrn> cloud_connection: Could not connect to nRF Cloud, readiness timeout
[00:11:37.077,209] <wrn> poc_app: Failed to establish connection to nRF Cloud
[00:11:42.077,301] <inf> poc_app: Connecting over LTE..
[00:11:42.086,578] <inf> poc_app: LTE connection established. Connecting to nRF Cloud.
[00:12:16.288,696] <inf> cloud_connection: Connection to nRF Cloud established and ready