WDT and Error Handler

RE: WDT and Error Handler

jlz — Wed, 14 Nov 2018 13:28:27 GMT

one more thing is that the callback is never triggered so I'm not sure if the reboot are related to a WDT timeout. Even so this software reset is only triggered when WDT are enabled.

RE: WDT and Error Handler

jlz — Wed, 14 Nov 2018 13:11:23 GMT

So I sent a question 2 months ago stating that feeding was not used. According to my investigation it behaved normally without kicking the dog. I assumed that this was handled in zephyr and basically because it is best handled in the OS. Adding feeding everywhere in custom code where you think you would have an issue does not make sense. The aim is not to change anything the the Zephyr and keeping it clean and more easy to maintain. I strongly recommend that you implement this for zephyr.

I have to test this before we know if it works.

RE: WDT and Error Handler

angl — Wed, 14 Nov 2018 12:29:57 GMT

Hi,

[quote userid="6656" url="~/f/nordic-q-a/38926/wdt-and-error-handler/156907"]My investigation reading the TRM and the sample code and also test the WDT it seems that feeding is not needed or part of the OS when the WDT function are enabled in Zephyr.[/quote]

This is not true. You definitely need to feed the watchdog after you enable it. Otherwise, it will reset your system. And you actually observe this:

[quote userid="6656" url="~/f/nordic-q-a/38926/wdt-and-error-handler/156907"]When my code runs it last for 2-4 hours and if during this time, if I hang a process for 5 seconds it will reboot. And this is expected behavior (I use 5 second timeout).[/quote]

When you do not hang a process, your code runs but as you stated it:

[quote userid="6656" url="~/f/nordic-q-a/38926/wdt-and-error-handler/156907"]3. sleeps most of the time and just doing sporadic activity's.[/quote]

and since you use the WDT_OPT_PAUSE_IN_SLEEP option, the watchdog timer does not count down during the CPU sleep periods. Therefore, the watchdog will not timeout after 5 seconds of the overall time, but after 5 seconds of the cumulated activity time of the CPU (this is the time when the watchdog is instructed to work in your configuration). And this is not only the time in which your application tasks are executed but also any interrupts that are handled in your system, scheduling etc. Hence, what you describe as a "sporadic reset" after 2-4 hours is actually the reset caused by the watchdog because it was not fed. You can confirm this by doing some indication in the watchdog handler, for instance by changing a pin state in this handler and monitoring of this pin with a logic analyzer (please note that the handler is fired only two cycles of the 32.768 kHz clock before the watchdog reset is done, so it is a very short time to do anything; if you for example turn a LED in the handler, you probably won't be able to notice it is turning on because the reset will shortly turn it off).

And the above is the answer to your question:

[quote userid="6656" url="~/f/nordic-q-a/38926/wdt-and-error-handler/156452"]So back to my OP question. Why do the system reboot within 2-4h?[/quote]

What to do to prevent this reboots? Feed the watchdog. Where to do it? It highly depends on your application. If you for instance have a task that mostly sleeps but will be woken up at least once per 5 seconds (or what you decide to configure the watchdog to) of the CPU activity time, it will be sufficient to place the feeding there. If the situation is more complicated and there is a possibility that only some event handler is executed without involving the task for a longer time (still, the cumulated activity time of the CPU), you'll need to add the feeding also in this handler. The key thing is to feed the watchdog only as a confirmation that the vital parts of your application are executed as expected.

RE: WDT and Error Handler

run_ar — Wed, 14 Nov 2018 11:12:19 GMT

You need to feed the Watchdog. Since you are only doing sporadic activities, you are sleeping most of the time, which means the Watchdog is halted. Since the watchdog only runs while the cpu is running it can take hours before it resets. The time it takes before it resets depends on how much time your application spends in "sleep". If you had a cpu intensive application the watchdog would reset your device much sooner. I would recommend setting up a task to feed the watchdog every 4th second or something like that.

RE: WDT and Error Handler

jlz — Mon, 12 Nov 2018 14:04:23 GMT

1. yes

2. no I do not and this information was proved in the OP.

3. sleeps most of the time and just doing sporadic activity's.

My investigation reading the TRM and the sample code and also test the WDT it seems that feeding is not needed or part of the OS when the WDT function are enabled in Zephyr. When my code runs it last for 2-4 hours and if during this time, if I hang a process for 5 seconds it will reboot. And this is expected behavior (I use 5 second timeout). If there was any issue with feeding it would reset within 5 seconds?

RE: WDT and Error Handler

run_ar — Mon, 12 Nov 2018 13:49:17 GMT

Hi,

I was asked to take a look at this thread as Håkon is out of office for some time. It's a bit hard to follow, so can you please confirm:

1. You have enabled the watchdog. And configured it to WDT_OPT_PAUSE_IN_SLEEP. So it should not run while the cpu sleeps.

2. You are not feeding it?

3. This leads to sporadic resets (i.e. resets after 1-3hours when the application does nothing) ?

If you enable the watchdog but do not feed it, it will eventually reset the device. When the reset happens depends on your configuration. If you configured it to pause while the CPU is sleeping it will still run while the zephyr os runs in the background at a higher interrupt level. So your application will be asleep (no running tasks), but the watchdog will run while the OS operations run. So the varying reset time will depend on how much activity you have in the os, even if your applications does nothing. Because of this you need to setup a recurring task that feeds the watchdog before it times out.

RE: WDT and Error Handler

jlz — Mon, 12 Nov 2018 13:03:36 GMT

I can but please do the verification on your end. I have no time to test your platform.

RE: WDT and Error Handler

Carles — Mon, 12 Nov 2018 13:01:36 GMT

Can you try the following patch? This will try to reduce the latency of Watchdog interrupts

diff --git a/drivers/watchdog/wdt_nrfx.c b/drivers/watchdog/wdt_nrfx.c
index 4297789c6a..2200441ae7 100644
--- a/drivers/watchdog/wdt_nrfx.c
+++ b/drivers/watchdog/wdt_nrfx.c
@@ -197,7 +197,7 @@ static int init_wdt(struct device *dev)
                return -EBUSY;
        }

-       IRQ_CONNECT(CONFIG_WDT_NRF_IRQ, CONFIG_WDT_NRF_IRQ_PRI,
+       IRQ_DIRECT_CONNECT(CONFIG_WDT_NRF_IRQ, 0,
                    nrfx_isr, nrfx_wdt_irq_handler, 0);
        irq_enable(CONFIG_WDT_NRF_IRQ);

RE: WDT and Error Handler

jlz — Thu, 08 Nov 2018 10:24:16 GMT

I did, 3 weeks ago.

RE: WDT and Error Handler

Carles — Thu, 08 Nov 2018 10:09:03 GMT

[quote userid="24487" url="~/f/nordic-q-a/38926/wdt-and-error-handler/156378"]So please try to setup watchdog with this option. If the problem will occurs after this change we try to prepare simple example together (extraction of affected code, based on zephyr and next one based on nrfx project to exclude influence of the zephyr RTOS) and pass it to Nordic engineers to look deply in peripheral's and CPU behaviour.[/quote]

jlz, please follow the advice above

RE: WDT and Error Handler

jlz — Thu, 08 Nov 2018 09:42:51 GMT

I do not use wdt_setup(system_wdt_dev, 0) I use wdt_setup(system_wdt_dev, WDT_OPT_PAUSE_IN_SLEEP).

see commit :

Even if I was clear how the configuration works you still want me to experiment with something that do not make sense. So I did test your suggestion and it fails because of my explanation above. The bit configuration are bitmask so the drivers knows what to configure. Bit 0 controls the sleep wdt and it should be set to 0. And this are done with wdt_setup(system_wdt_dev, WDT_OPT_PAUSE_IN_SLEEP). With your experiment it does not set anything (bit mask set 0 -> setting nothing). This means that wdt for sleep will be set too 1 (default configuration) so we will reboot every-time we going to sleep.

So back to my OP question. Why do the system reboot within 2-4h?

RE: WDT and Error Handler

jlz — Thu, 08 Nov 2018 09:28:05 GMT

I just read the thread again because I reference on older comments but they are gone. Have you deleted my old comments?

RE: WDT and Error Handler

jlz — Thu, 08 Nov 2018 09:19:53 GMT

I see the issue here, some else have comment on this related to the last comment from me that was a request to change my code to something I already now will fail (explained above). I do not use this and was a request to show the code snippet on a change that Nordic (hakan) wanted to change that was wrong in my mind.

I use wdt_setup(system_wdt_dev, WDT_OPT_PAUSE_IN_SLEEP) and have done this for all of my test.

I only see one way forward and that is Nordic test the WDT in Zephyr and comeback when it is veryfied that it works and if so provide the solution for this.

Here is some more info that was not part of the code snippet:

#define SYSTEM_WDT_WINDOW_MAX 5000 // ms
#define SYSTEM_WDT_WINDOW_MIN 0 // ms

RE: WDT and Error Handler

kl-cruz — Wed, 07 Nov 2018 18:44:09 GMT

Hi,

Your code looks good.

You set correct reset SOC flag - nRFs support only this mode.

wdt_config.flags = WDT_FLAG_RESET_SOC;

You set window.max value, I assume that your min value (SYSTEM_WDT_WINDOW_MIN) is zero. nRF hardware support basic situation when feed window is constant (starts at time 0 and expire in future - max).

wdt_config.window.min = SYSTEM_WDT_WINDOW_MIN;
wdt_config.window.max = SYSTEM_WDT_WINDOW_MAX;

Callback is nice feature, but you have to be sure that you are doing things very quickly. You have two 32kHz cycles (http://infocenter.nordicsemi.com/index.jsp?topic=%2Fcom.nordic.infocenter.nrf52832.ps.v1.1%2Fwdt.html&cp=2_1_0_39_2&anchor=concept_b4n_xzw_sr). The code is executing with 64MHz frequency (I assume situation without jumps/branches with flash wait states equals zero, clocks are synchronized - lucky we) so 1953 full CPU cycles, but in real situation less.

You are calling setup function with 0 flag. In this case watchdog is kept running while the CPU is sleeping and is kept running when debugger is attached.

if (wdt_setup(system_wdt_dev, 0) < 0) {

It is the place where I can see (eventually) potential problem. Maybe when watchdog is still running you miss the feed window in one/two/four hours. I can't confirm that. I just assume. You can connect logic analyzer to your GPIOs and try to trace situations (measure feed time etc) to check this.

You can configure to pause watchdog while cpu is sleeping using flag previously recommended by Hakon. It is WDT_OPT_PAUSE_IN_SLEEP which set 1 at position zero. Watchdog driver checks if there is one at this position and set correct behaviour of the peripheral:

https://github.com/zephyrproject-rtos/zephyr/blob/master/drivers/watchdog/wdt_nrfx.c#L42

Please try options when watchdog is paused when CPU is sleeping:

wdt_setup(system_wdt_dev, WDT_OPT_PAUSE_IN_SLEEP)

I tried to find any errata entries about watchdog. There are two MCUs affected by WDT problems with increasing current consumption, but threre is no information about unexpected resets.

So please try to setup watchdog with this option. If the problem will occurs after this change we try to prepare simple example together (extraction of affected code, based on zephyr and next one based on nrfx project to exclude influence of the zephyr RTOS) and pass it to Nordic engineers to look deply in peripheral's and CPU behaviour.

Please tell me about your observations afterall. Thanks!

RE: WDT and Error Handler

jlz — Wed, 07 Nov 2018 13:22:04 GMT

requested code snippet:

    system_wdt_dev = device_get_binding(CONFIG_WDT_0_NAME);

    // Reset SoC when watchdog timer expires.
    wdt_config.flags = WDT_FLAG_RESET_SOC;

    // Expire watchdog after x milliseconds.
    wdt_config.window.min = SYSTEM_WDT_WINDOW_MIN;
    wdt_config.window.max = SYSTEM_WDT_WINDOW_MAX;

    // Set up watchdog callback. Jump into it when watchdog expired.
    wdt_config.callback = system_wdt_callback;

    wdt_channel_id = wdt_install_timeout(system_wdt_dev, &wdt_config);

    if (wdt_channel_id < 0) {

        return 1; // **** ERROR ****
    }

    if (wdt_setup(system_wdt_dev, 0) < 0) {

        return 1; // **** ERROR ****
    }

    return 0; // **** OK ****

RE: WDT and Error Handler

Carles — Wed, 07 Nov 2018 13:21:33 GMT

Can you please post a code snippet with the code you have tried?

RE: WDT and Error Handler

jlz — Wed, 07 Nov 2018 11:29:06 GMT

So back to my OP question. Why do the system reboot within 2-4h?

RE: WDT and Error Handler

Hakon — Tue, 06 Nov 2018 09:47:01 GMT

https://github.com/zephyrproject-rtos/zephyr/blob/master/include/misc/util.h. Look up the definition of BIT(). And then look at this www.wolframalpha.com/.../. Then maybe it will be more apparent that BIT(0) and 0 is not the same. I will see if I have time to reproduce your issue later, but I'm a little busy right now.

RE: WDT and Error Handler

jlz — Tue, 06 Nov 2018 09:24:06 GMT

its a mask bit so it will write 0. Can you please test this yourself and see if it working. Note that it will take 2-4 hours to trigger this issue.

RE: WDT and Error Handler

Hakon — Tue, 06 Nov 2018 09:21:41 GMT

Because it's not 0. It's BIT(0), i.e. 1. So calling wdt_setup(system_wdt_dev, 0) is not the same as calling wdt_setup(system_wdt_dev, WDT_OPT_PAUSE_IN_SLEEP).

RE: WDT and Error Handler

jlz — Tue, 06 Nov 2018 09:16:10 GMT

yes

RE: WDT and Error Handler

Hakon — Tue, 06 Nov 2018 09:14:19 GMT

Are you referring to this?

#define WDT_OPT_PAUSE_IN_SLEEP BIT(0)

in github.com/.../watchdog.h

RE: WDT and Error Handler

jlz — Mon, 05 Nov 2018 16:19:46 GMT

Yes I do, WDT_OPT_PAUSE_IN_SLEEP = 0. Can you please check this issue!

RE: WDT and Error Handler

Hakon — Mon, 05 Nov 2018 14:03:20 GMT

Have you tried to call wdt_setup() like this instead?

wdt_setup(system_wdt_dev, 0)

This should keep the watchdog from pausing in sleep mode.

RE: WDT and Error Handler

jlz — Wed, 31 Oct 2018 15:08:10 GMT

So I have tested this and it does not work. First is that I uses this setting in the code (basically had this from the beginning):

wdt_setup(system_wdt_dev, WDT_OPT_PAUSE_IN_SLEEP)

I when change the code to run this every time before enter sleep but it still do not work.

Random reboot ones within 2-3h