nRF Cloud FOTA stuck in QUEUED

[RossJ]

A couple of days ago I successfully upgraded 10 of our nRF9160 devices using FOTA in nRF Cloud (MQTT). Several devices didn't upgrade and remained in status QUEUED. I've since been unable to get any further devices to upgrade. All queued devices are online (as reported by nRF Cloud) and are actively communicating with our own server via LTE-M. The devices are all the same hardware and software configuration. I've since cancelled/deleted/recreated the upgrades (a couple of times) but always with the same outcome. Several devices have remained in queued state for over 36 hours!

[helsing]

Hi Ross,

Just letting you know we have started looking into this.

[helsing]

A couple of days ago I successfully upgraded 10 of our nRF9160 devices using FOTA in nRF Cloud (MQTT).

How many devices do you have in total that you attempted to upgrade?

Several devices didn't upgrade and remained in status QUEUED.

How many devices are in this status? Do you have access to these devices? Would you be able to provide any logs or traces?

Are you able to successfully upgrade any test devices in the lab?

[RossJ]

I have resolved the issue. It turned out that the nRFCloud library was attempting to read own IP address via modem info (AT+CGDCONT?). However, the buffer size set by CONFIG_MODEM_INFO_BUFFER_SIZE (default: 128) is insufficient in my case. The result was a failure to fully connect to the cloud and post its shadow. nRFCloud still showed my devices as 'online', just without shadow updates or working FOTA!

I suspect this has just become an issue when AT+CGDCONT? began including an IPV6 address as well as the IPV4 address. This pushes it over the buffer limit. I have done numerous FOTA updates over the past year and this has only occurred recently. Even so, a couple of our devices have still managed to update, so it seems the IPV6 address may still not always be present for me. This has presumably been brought on by changes in the Telstra (Australia) network.