Issue when application that uses TASK WDT gets a FOTA with an application that uses simple WDT

RE: Issue when application that uses TASK WDT gets a FOTA with an application that uses simple WDT

Einar Thorsrud — Fri, 17 Mar 2023 15:17:49 GMT

Hi,

In principle, yes. Except if you are able to confirm your new firmware before the 10 seconds timeout, and then let the WDT timeout. If that is possible, your new application would start and the WDT will not be enabled, so that it can be configured as you please. This approach may not be possible or desirable, though.

RE: Issue when application that uses TASK WDT gets a FOTA with an application that uses simple WDT

Giannis Anastasopoulos — Thu, 16 Mar 2023 17:33:34 GMT

Based on a test that I just did, it seems that the HW watchdog is configured from the application that is running the Task WDT to expire at 10s. In the app that I am trying to FOTA, if I feed the WDT in less than 10s then the application does not reset.

So based on that result, it seems that due to the fact that the watchdog is set from the first application to expire in 10s and since the WDT cannot be stopped due to system reset, the WDT is running with the old configuration even though I am setting it up again using the

wdt_install_timeout(wdt, &wdt_config);

This means that the only way that my settings will ever have impact is if the device was reset by the reset pin or powercycled, in which case the device would revert back to the old firmware though, since the new firmware is not yet confirmed.

RE: Issue when application that uses TASK WDT gets a FOTA with an application that uses simple WDT

Giannis Anastasopoulos — Thu, 16 Mar 2023 16:49:53 GMT

In an attempt to rule out the possibility of a channel different than the first one being used in the initial application, in the FOTA application I am doing this:

    int wdt_channel_id = 0;
    while (wdt_channel_id >= 0) {
        max_wdt_channel_id = wdt_channel_id;
        wdt_channel_id = wdt_install_timeout(wdt, &wdt_config);
        LOG_INF("WDT Channel %d set", wdt_channel_id);
    }
    LOG_INF("MAX channel ID: %d", max_wdt_channel_id);

And I feed all the channels like this:

    for(int i = 0; i <= max_wdt_channel_id; i++){
        LOG_INF("Feeding watchdog, channel: %d", i);
        wdt_feed(wdt, i);
    }

But my device still resets due to DOG reset

18:38:03.090 -> *** Booting Zephyr OS build v2.6.0-rc1-ncs1  ***
18:38:03.136 -> [00:00:00.240,051] [0m<inf> IMS: IMS dispatch thread active[0m
18:38:03.136 -> [00:00:00.251,159] [0m<inf> fs_nvs: 3 Sectors of 4096 bytes[0m
18:38:03.136 -> [00:00:00.251,190] [0m<inf> fs_nvs: alloc wra: 2, 5c0[0m
18:38:03.136 -> [00:00:00.251,190] [0m<inf> fs_nvs: data wra: 2, 514[0m
18:38:03.184 -> Reset caus[00:00:00.343,994] [0m<inf> DEVICE_SETTINGS: NVM now accepting new settings[0m
18:38:03.469 -> e(s):
18:38:03.469 -> 	SREQ
18:38:03.469 -> [00:00:00.354,461] [0m<inf> regular_reset: Handler active[0m
18:38:03.469 -> [00:00:00.354,858] [0m<inf> IMS_STATS: Stats listener active[0m
18:38:03.469 -> [00:00:00.355,041] [0m<inf> IMS_STATS: IMS Stats listener active[0m
18:38:03.469 -> [00:00:00.355,255] [0m<inf> WDT: WDT Channel 0 set[0m
18:38:03.469 -> [00:00:00.355,285] [0m<inf> WDT: WDT Channel 1 set[0m
18:38:03.469 -> [00:00:00.355,316] [0m<inf> WDT: WDT Channel 2 set[0m
18:38:03.469 -> [00:00:00.355,316] [0m<inf> WDT: WDT Channel 3 set[0m
18:38:03.469 -> [00:00:00.355,346] [0m<inf> WDT: WDT Channel 4 set[0m
18:38:03.469 -> [00:00:00.355,346] [0m<inf> WDT: WDT Channel 5 set[0m
18:38:03.469 -> [00:00:00.355,377] [0m<inf> WDT: WDT Channel 6 set[0m
18:38:03.469 -> [00:00:00.355,377] [0m<inf> WDT: WDT Channel 7 set[0m
18:38:03.469 -> [00:00:00.355,407] [0m<inf> WDT: WDT Channel -12 set[0m
18:38:03.469 -> [00:00:00.355,407] [0m<inf> WDT: MAX channel ID: 7[0m
18:38:03.469 -> [00:00:00.355,438] [0m<inf> WDT: Feeding watchdog, channel: 0[0m
18:38:03.469 -> [00:00:00.355,438] [0m<inf> WDT: Feeding watchdog, channel: 1[0m
18:38:03.469 -> [00:00:00.355,468] [0m<inf> WDT: Feeding watchdog, channel: 2[0m
18:38:03.469 -> [00:00:00.355,468] [0m<inf> WDT: Feeding watchdog, channel: 3[0m
18:38:03.469 -> [00:00:00.457,611] [0m<inf> WDT: Feeding watchdog, channel: 4[0m
18:38:03.469 -> [00:00:00.457,641] [0m<inf> WDT: Feeding watchdog, channel: 5[0m
18:38:03.469 -> [00:00:00.457,672] [0m<inf> WDT: Feeding watchdog, channel: 6[0m
18:38:03.469 -> [00:00:00.457,702] [0m<inf> WDT: Feeding watchdog, channel: 7[0m
18:38:03.469 -> [00:00:00.457,916] [0m<inf> IMS_WS2812: Listener active[0m
18:38:03.469 -> [00:00:00.458,190] [0m<inf> IMS_BATT: Listener active[0m
18:38:03.469 -> [00:00:00.458,282] [0m<inf> WDT: Feeding watchdog, channel: 0[0m
18:38:03.469 -> [00:00:00.458,282] [0m<inf> WDT: Feeding watchdog, channel: 1[0m
18:38:03.469 -> [00:00:00.458,312] [0m<inf> WDT: Feeding watchdog, channel: 2[0m
18:38:03.469 -> [00:00:00.458,374] [0m<inf> WDT: Feeding watchdog, channel: 3[0m
18:38:03.469 -> [00:00:00.458,404] [0m<inf> WDT: Feeding watchdog, channel: 4[0m
18:38:03.469 -> [00:00:00.458,435] [0m<inf> WDT: Feeding watchdog, channel: 5[0m
18:38:03.469 -> [00:00:00.458,435] [0m<inf> WDT: Feeding watchdog, channel: 6[0m
18:38:03.469 -> [00:00:00.458,435] [0m<inf> WDT: Feeding watchdog, channel: 7[0m
... (not important device init)
18:38:14.319 -> ⸮*** Booting Zephyr OS build v3.1.99-ncs1  ***
18:38:14.549 -> I: Starting bootloader
18:38:14.596 -> I: Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x3
18:38:14.596 -> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
18:38:14.596 -> I: Boot source: none
18:38:14.596 -> I: Swap type: revert
18:38:15.298 -> I: Starting swap using move algorithm.
18:38:15.298 -> I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
18:38:55.727 -> I: Bootloader chainload address offset: 0x10000
18:38:55.727 -> I: Jumping to the first image slot
[00:00:00.457,183] [0m<inf> spi_nor: W25Q16JV: SFDP v 1.5 AP ff with 1 PH[0m
18:38:56.522 -> [00:00:00.457,214] [0m<inf> spi_nor: PH0: ff00 rev 1.5: 16 DW @ 80[0m
18:38:56.522 -> *** Booting Zephyr OS build v3.1.99-ncs1  ***
18:38:56.522 -> [00:00:00.471,466] [0m<inf> spi_nor: W25Q16JV: 2 MiBy flash[0m
18:38:56.522 -> [00:00:00.475,799] [0m<inf> mcuboot_util: Swap type: none[0m

18:38:56.522 -> 

18:38:56.522 -> Reset cause(s):
18:38:56.522 -> 	DOG

As you can see I am feeding all the HW channels right at boot up, but the device is still resetting after a few seconds. And the reset reason is DOG and there is no hardfault there

RE: Issue when application that uses TASK WDT gets a FOTA with an application that uses simple WDT

Einar Thorsrud — Thu, 16 Mar 2023 15:19:49 GMT

[quote user="Giannis Anastasopoulos"]What I am trying to do is update the device with an application that is using the standard HW WDT alone.[/quote]

Ah, I see. I misunderstood the situation. The fundamentals remain the same, though.

[quote user="Giannis Anastasopoulos"]What I am doing in the new application is to simply feed the WDT as the first thing in the main.[/quote]

That should normally be good enough. What is the reload-value of the WDT (the old configuration is the relevant here, as that is what is being used)? That will let us know how much time you have.

Also, is there by any change more than one channel enabled?

If we know these two things, then we know what you need to do in the new application in order to continue to feed the watchdog.

It would also be relevant to know if there are any other parts of the system (typically bootlaoder(s)) that enables the WDT? If not, you would at least be able to re-configure it to suit your new needs after a WDT reset.

[quote user="Giannis Anastasopoulos"]Is there a way to reset or disable the watchdog on the original application with the TASK WDT, right before doing the reset to boot to the new application?[/quote]

No. There is no support for disabling the Task WDT. And more importantly, you cannot disable the HW WDT (which you write is enabled here as the fallback) in any other way than a reset of some sort which is not a soft reset or CPU lockup. The only way to disable it would be to trigger a reset of a form that disables it (like a WDT reset by not feeding it).

RE: Issue when application that uses TASK WDT gets a FOTA with an application that uses simple WDT

Giannis Anastasopoulos — Thu, 16 Mar 2023 14:33:55 GMT

Einar Thorsrud

Thanks a lot for your response. My issue is this:

My original application is using the TASK WDT with the HW WDT enabled as a fallback.

What I am trying to do is update the device with an application that is using the standard HW WDT alone.

What I am doing in the new application is to simply feed the WDT as the first thing in the main.

But this still does not seem to have an impact as the device resets and the reset reason is still DOG.

Is there a way to reset or disable the watchdog on the original application with the TASK WDT, right before doing the reset to boot to the new application?

RE: Issue when application that uses TASK WDT gets a FOTA with an application that uses simple WDT

Einar Thorsrud — Thu, 16 Mar 2023 13:18:27 GMT

Hi,

The HW WDT is (deliberately) quite limited with regards to how it can be disabled. Once enabled, it will continue to run and the configuration cannot be changed, except after a reset of the WTD peripheral. And that is not reset during a normal soft reset (see Reset behavior).

So if the currently running firmware (you have based on 1.6) is using the WDT, you will need to continue to feed it. As you are getting resets with DOG as reason in the RESETREAS, that means it has not been fed fast enough, timed out, and reset the device.

The Task WDT has the option of using a physical WDT as fall back, and so it also must feed it. So it could be that you can get away with just increasing the frequency of the WDT feeding in the Task WDT configuration. You adjust that with a combination of CONFIG_TASK_WDT_MIN_TIMEOUT and CONFIG_TASK_WDT_HW_FALLBACK_DELAY. Alternatively, add some special handling to ensure it is feed often enough. But of course, make sure you don't just feed it all the time regardless of the state of the device, as if you do, there will not be much point of the WDT.)

As a side note, perhaps an alternative could be to keep using the physical WDT peripheral in the nRF and not the task WDT library at all? The WDT peripheral in the nRF has 8 channels, so you can monitor 8 tasks separately with that if you like (though again, you cannot change the configuration registers once it is started, so it may be difficult if you do not already use it like this in the existing firmware). Remember that the task WDT is a zephyr library, and if things go really haywire it may not be able to save you in the same way as a physical WDT timer can.

Lastly, if you need to change the configuration of the HW WDT, it is possible to first upload via DFU some firmware that does not start it but lets it time out and reset (assuming you do not start it in for instance an immutable bootloader - if you do, it can never change). Then, after the WDT reset the WDT is not running, and if your firmware does not start it, well - it is not started, and you can configure it as you like or not use it at all.