FOTA Doom Loop After Failed Download

RE: FOTA Doom Loop After Failed Download

Sigurd Hellesvik — Thu, 09 Jan 2025 13:04:19 GMT

That makes sense to me, thanks for the elaboration!

RE: FOTA Doom Loop After Failed Download

Mike Granby — Thu, 09 Jan 2025 12:51:12 GMT

A watchdog is great and I have it enabled, but it would not be triggered by firmware that is continually re-attempting a download as it would not be able to distinguish between that failure mode and, say, the mere lack of cellular signal. The watchdog can only catch low-level failures or those that are detected by the logic that kicks it. In this case, I have added specific code to spot repeated download failures and to reboot the device, but that required a priori knowledge of the problem.

RE: FOTA Doom Loop After Failed Download

Sigurd Hellesvik — Thu, 09 Jan 2025 12:44:35 GMT

Hi Mike,

I will continue to help with this ticket.

Just want you to know that we have not forgotten this ticket. Devs are not done looking into it yet, but we are on it.

A comment from me on previous things you said. Specifically:

"A reboot fixes it, but unrecoverable FOTA failures are bad news..."

and "It’s the fact that it gets stuck in a loop trying to recover."

I definitely agree that the device should not enter a loop that it can not escape. However, software can in general introduce deadlocks, and it is hard to guarantee 100% that you can never get into one of those.

However, it is a lot easier to guarantee a stable state on a reboot. This is why we have the Watchdog hardware peripheral. If set up properly, the Watchdog should reboot the device if something puts it into a deadlock. Is this a solution you have tried or considered?

RE: FOTA Doom Loop After Failed Download

Hung Bui — Tue, 17 Dec 2024 13:18:58 GMT

Hi Mike,
I'm waiting for R&D team to take a look. The initial response is that they suspect it's the dfu_target library causing the problem as it's where the offset comes from.

RE: FOTA Doom Loop After Failed Download

Mike Granby — Mon, 16 Dec 2024 15:00:30 GMT

I’m not sure what caused the initial error. I know I was updating the firmware file on the server, but the point is not the initial error. It’s the fact that it gets stuck in a loop trying to recover. Obviously a FOTA failure means a potential truck roll, and that’s bad.

RE: FOTA Doom Loop After Failed Download

Hung Bui — Mon, 16 Dec 2024 12:58:18 GMT

Hi Mike,
I will take a look at the issue. But could you let me know what happen before the Error in recv(), errno 11 ?
Could you tell how did you test when did the issue occur and if you see this error 11 on every test ?