FOTA triggered with very long delay

RE: FOTA triggered with very long delay

dejans — Fri, 18 Jul 2025 12:48:00 GMT

Hi,

[quote user="teba99"]I can try the unmodified multi_service example, but it may be a while before I have any results because I have limited cycles and the problem can't be readily reproduced.[/quote]

You can try multi_service sample to check if the same issue is present with unmofied sample as in the sample with your modifications. Please let me know if you are able to reproduce the issue with unmodified multi_service sample.

Best regards,
Dejan

RE: FOTA triggered with very long delay

teba99 — Fri, 18 Jul 2025 10:37:32 GMT

Hi Dejan,

We use MQTT. We have not used AT%XMONITOR yet.

I can try the unmodified multi_service example, but it may be a while before I have any results because I have limited cycles and the problem can't be readily reproduced.

Going back to my initial question, I was trying to understand if there may be a known issue with delayed FOTA. But it seems like you are not aware of any pending issues and the occurrence seems rare, so right now it's not a very high priority for us.

Thanks!

RE: FOTA triggered with very long delay

dejans — Wed, 16 Jul 2025 15:38:51 GMT

Hi,

Do you use CoAP or MQTT?
Can you test unmodified multi_service sample on your affected board to see if you can replicate the problem?
Which result do you get when using AT%XMONITOR?

Best regards,
Dejan

RE: FOTA triggered with very long delay

teba99 — Wed, 16 Jul 2025 11:43:57 GMT

Hi Dejan,

We noticed the issue about 3 weeks ago -- the update to that device was listed as "QUEUED" for about 2 weeks. Then, a week ago, the device suddenly updated. As far as we know, nothing on our side changed, except that the device was power-cycled three or four times during the 2 weeks for different reasons. The last power cycle seemed to have triggered the OTA.

The device was working as expected all the time, sending data to the cloud.

At the core, our application is build on the multi-service example. The OTA job is created and deployed in the nRF Cloud UI. After deploying, the OTA usually is executed by a device within a minute. We have done about 100 OTA updates this way, and they worked fine. It's just this one device so far that took much longer than expected.

I've attached two screenshots:

1. The OTA job deployed on Jun 13

2. The OTA job on Jul 7 it finally succeeded (note: We did another OTA the following day on Jul 8 as a test, that succeeded quickly, as expected).

Best,

-- Terrence

RE: FOTA triggered with very long delay

dejans — Wed, 16 Jul 2025 08:50:08 GMT

Hi Terrence,

When have you noticed OTA update problem on this particular device?

With regard to OTA, has device worked as expected before and after the issue appeared?

Can you provide some details about your OTA procedure?

Best regards,
Dejan

RE: FOTA triggered with very long delay

teba99 — Mon, 14 Jul 2025 21:46:17 GMT

Hi Dejan,

We'll try increasing the modem buffer.

[quote userid="111786" url="~/f/nordic-q-a/122839/fota-triggered-with-very-long-delay/542424"]Can you provide more information about turning-off and power-cycling? Is there any relationship between these 2 actions?
Who is turning the devices on/off and power-cycling them?[/quote]

The devices can be turned off or power-cycled for different reasons, by the operator. There should be no difference in the result ... once the device starts running again it connects to nRF Cloud and should execute the OTA, if one is pending.

[quote userid="111786" url="~/f/nordic-q-a/122839/fota-triggered-with-very-long-delay/542424"]

Will you be able to provide a log?

[/quote]

As I said, the log has no information related to the OTA. It just shows the device connecting and then going about its business (collecting and sending data to the cloud). I don't think it will be useful.

[quote userid="111786" url="~/f/nordic-q-a/122839/fota-triggered-with-very-long-delay/542424"]

When did this issue start occurring? Was it a sudden event?

[/quote]

That's the weird thing. We are issuing OTA updates to devices often. It works flawlessly almost all the time, except for this one device. It is the same device hardware and software as all the other devices and it is otherwise behaving identical to the other devices -- except for the OTA.

[quote userid="111786" url="~/f/nordic-q-a/122839/fota-triggered-with-very-long-delay/542424"]Did you see this issue only on a single device? Is this issue reproducible on the same device or on other devices?[/quote]

Currently only once and only on one device.

Thanks,
-- Terrence

RE: FOTA triggered with very long delay

dejans — Mon, 14 Jul 2025 16:30:24 GMT

Hi Terrence,

[quote user="teba99"]The symptom is the same as we experience. Maybe we'll try to increase CONFIG_MODEM_INFO_BUFFER_SIZE and see if that helps.[/quote]

It would be good to see if increasing buffer size resolves the issue.

[quote user="teba99"]When the OTA finally did happen, the device was turned off for a few hours, then turned on, and then the OTA suddenly occurred. However, the device was power-cycled before that already several times and the OTA did NOT occur. We haven't found any pattern or trigger.[/quote]

Can you provide more information about turning-off and power-cycling? Is there any relationship between these 2 actions?
Who is turning the devices on/off and power-cycling them?

[quote user="dejans"]Can you clarify if you have device log or not? Device log might be helpful. Can you share the log if you have it?[/quote]

Will you be able to provide a log?

[quote user="dejans"]Can you provide information when this delaying FOTA issue occured?[/quote]

When did this issue start occurring? Was it a sudden event?

[quote user="dejans"]Did this instance of FOTA update delay happen only once and on a single device?[/quote]

Did you see this issue only on a single device? Is this issue reproducible on the same device or on other devices?

Best regards,
Dejan

RE: FOTA triggered with very long delay

teba99 — Mon, 14 Jul 2025 08:01:13 GMT

Hi dejans

Thanks for the link. We are capturing device logs on the serial port and in the nRF cloud, but there are no entries related to the OTA update. It's like the device doesn't see that an OTA is pending, and there are no errors or warnings either. The device just sits there and operates fine, but never sees the OTA.

When the OTA finally did happen, the device was turned off for a few hours, then turned on, and then the OTA suddenly occurred. However, the device was power-cycled before that already several times and the OTA did NOT occur. We haven't found any pattern or trigger.

[quote userid="111786" url="~/f/nordic-q-a/122839/fota-triggered-with-very-long-delay/542096"]nRF Cloud FOTA stuck in queued.[/quote]

The symptom is the same as we experience. Maybe we'll try to increase CONFIG_MODEM_INFO_BUFFER_SIZE and see if that helps.

Thanks,
-- Terrence

RE: FOTA triggered with very long delay

dejans — Thu, 10 Jul 2025 11:33:35 GMT

Hi,

Maybe you have similar issue as shown in nRF Cloud FOTA stuck in queued.

Did this instance of FOTA update delay happen only once and on a single device?

[quote user=""]But in the device log we could see the device is not getting the FOTA notification and therefore does not start the download. The FOTA just stays as "QUEUED" but never changes to "DOWNLOADING".[/quote][quote user=""]The problem is very difficult to reproduce and we don't have logs.[/quote]

Can you clarify if you have device log or not? Device log might be helpful. Can you share the log if you have it?

Can you provide information when this delaying FOTA issue occured?

Best regards,
Dejan