Device intermittent failures to connect to nRF Cloud MQTT broker (err: -111)

[teba99]

Hi,

Intermittently, after a reset/restart some of our devices are unable to connect to the nRF Cloud MQTT broker:

[00:00:09.235,626] <inf> app_cloud_connection: Network connectivity gained!
[00:00:10.236,114] <inf> app_cloud_connection: Network is ready
[00:00:10.236,145] <inf> app_cloud_connection: Connecting to nRF Cloud
[00:01:00.567,901] <wrn> app_application: Cloud not ready within 60 seconds, proceeding with app startup
[00:01:10.252,746] <err> nrf_cloud_transport: Could not connect to nRF Cloud MQTT Broker mqtt.nrfcloud.com, port: 45858. err: -111
[00:01:10.252,807] <inf> app_cloud_connection: Disconnecting from nRF Cloud
[00:01:10.257,385] <inf> app_cloud_connection: Could not connect to nRF Cloud

[00:01:10.414,062] <inf> app_cloud_connection: Retrying nRF Cloud connection in 30 seconds...

Notes:

• This affects multiple devices.
• The error is intermittent. The devices connected fine just before the restart
• Once the error occurs, the device retries to connect but keeps getting this error indefinitely
• The device connection (LTE-M) appears to be working correctly as can be seen in the log, and we also see data flow in the SIM provider portal
• After another restart, the devices connect fine again.

Once a device is connected, we typically don't see this error.

It feels like there is a temporary network routing, firewall, or broker issue that prevents the device from connecting once. But the device network stack then gets stuck in a state where it now fails to reconnect again, even if the temporary problem went away. Restarting the device clears that state and allows the device to reconnect instantly.

Have you seen this before? Is there a workaround other then restarting the device?

Thanks

[Pascal Hdz]

Hello Terrence,

Why is the port pointing to 45858? It should be 8883. Please verify if the configuration variable

CONFIG_NRF_CLOUD_PORT

has been changed in your project.

Pascal.

[teba99]

Hi Pascal,

We were wondering the same, we are not making any changes to the port or CONFIG_NRF_CLOUD_PORT. In zephyr/autoconf.h I see 

#define CONFIG_NRF_CLOUD_PORT 8883

Thanks

[Pascal Hdz]

There might be something else in your application or configuration that might be changing this value.

Regards,

Pascal.

[teba99]

Hi Pascal,

I checked, we don't change the port anywhere. But I think the message about port 45858 may be misleading ... maybe it is an ephemeral local port on the device side?

In any case, I realized the problem is similar to one I reported a couple of months ago. I thought this was a different error, but it seems to be the same:  nRF9151 stuck in disconnected, unable to reconnect to LTE-M 

We'll investigate this as a combination of SIM/network behavior and trying to reset the TCP connection/modem stack.

Apologies. I think this post can be closed as a duplicate.

Update:

After some more debugging, I believe the issue here is different. The one I reported a few months ago is triggered specifically by SIM issues (e.g. deactivation), and as a follow-on effect may result in err: -111. 

In contrast, what I am reporting here is not triggered by SIM issues, but unprompted and seemingly random. "Could not connect to nRF Cloud MQTT Broker mqtt.nrfcloud.com, port: 45858. err: -111" seems to happen at any time, on any device we are using, independent of the SIM or cellular network.  In our device population it is currently happening multiple times a day across devices.

Any advice or insight you have is appreciated. 

Thanks