FOTA triggered with very long delay

[teba99]

Hi,

We are using a modified version of the multi-service sample for nRF9151. Generally, FOTA updates from the nRF Cloud work well.

Once we deploy the FOTA update in the nRF Cloud, if the device is online we see the device getting the FOTA notification and starting the update within about 1 to 5 minutes.

However, we have experienced an instance where there was a delay of several days until a device picked up the FOTA, despite running and being connected to the nRF Cloud. We tried re-creating/redeploying the update in the cloud and we power-cycled the device several times over the course of a few days. But in the device log we could see the device is not getting the FOTA notification and therefore does not start the download. The FOTA just stays as "QUEUED" but never changes to "DOWNLOADING".

After several days, we power-cycled the device again for a different reason, and now suddenly it picked up the OTA. We did not change anything on the device, so we think the issue lies with the nRF Cloud, e.g. the FOTA deployment suddenly came alive and triggered the device. 

The problem is very difficult to reproduce and we don't have logs. My question is: Is this a known issue that there sometimes can be a delay of many hours or even days between deploying an FOTA in the cloud and the FOTA actually being triggered on the device? Is there a recommendation to avoid this situation?

Thanks

[dejans]

Hi,

Maybe you have similar issue as shown in nRF Cloud FOTA stuck in queued.

Did this instance of FOTA update delay happen only once and on a single device?

But in the device log we could see the device is not getting the FOTA notification and therefore does not start the download. The FOTA just stays as "QUEUED" but never changes to "DOWNLOADING".

The problem is very difficult to reproduce and we don't have logs.

Can you clarify if you have device log or not? Device log might be helpful. Can you share the log if you have it?

Can you provide information when this delaying FOTA issue occured?

Best regards,
Dejan

[teba99]

Hi dejans 

Thanks for the link. We are capturing device logs on the serial port and in the nRF cloud, but there are no entries related to the OTA update. It's like the device doesn't see that an OTA is pending, and there are no errors or warnings either. The device just sits there and operates fine, but never sees the OTA.

When the OTA finally did happen, the device was turned off for a few hours, then turned on, and then the OTA suddenly occurred. However, the device was power-cycled before that already several times and the OTA did NOT occur. We haven't found any pattern or trigger.

dejans said:

nRF Cloud FOTA stuck in queued.

The symptom is the same as we experience. Maybe we'll try to increase CONFIG_MODEM_INFO_BUFFER_SIZE and see if that helps.

Thanks,
-- Terrence

[dejans]

Hi Terrence,

teba99 said:

The symptom is the same as we experience. Maybe we'll try to increase CONFIG_MODEM_INFO_BUFFER_SIZE and see if that helps.

It would be good to see if increasing buffer size resolves the issue.

teba99 said:

When the OTA finally did happen, the device was turned off for a few hours, then turned on, and then the OTA suddenly occurred. However, the device was power-cycled before that already several times and the OTA did NOT occur. We haven't found any pattern or trigger.

Can you provide more information about turning-off and power-cycling? Is there any relationship between these 2 actions?
Who is turning the devices on/off and power-cycling them?

dejans said:

Can you clarify if you have device log or not? Device log might be helpful. Can you share the log if you have it?

Will you be able to provide a log?

dejans said:

Can you provide information when this delaying FOTA issue occured?

When did this issue start occurring? Was it a sudden event?

dejans said:

Did this instance of FOTA update delay happen only once and on a single device?

Did you see this issue only on a single device? Is this issue reproducible on the same device or on other devices?

Best regards,
Dejan

[teba99]

Hi Dejan,

We'll try increasing the modem buffer.

dejans said:

Can you provide more information about turning-off and power-cycling? Is there any relationship between these 2 actions?
Who is turning the devices on/off and power-cycling them?

The devices can be turned off or power-cycled for different reasons, by the operator. There should be no difference in the result ... once the device starts running again it connects to nRF Cloud and should execute the OTA, if one is pending.

dejans said:

Will you be able to provide a log?

As I said, the log has no information related to the OTA. It just shows the device connecting and then going about its business (collecting and sending data to the cloud). I don't think it will be useful.

dejans said:

When did this issue start occurring? Was it a sudden event?

That's the weird thing. We are issuing OTA updates to devices often. It works flawlessly almost all the time, except for this one device. It is the same device hardware and software as all the other devices and it is otherwise behaving identical to the other devices -- except for the OTA.

dejans said:

Did you see this issue only on a single device? Is this issue reproducible on the same device or on other devices?

Currently only once and only on one device. 

Thanks,
-- Terrence