App timer stops working - need a power cycle to fix issue

[EduanShuda]

I am using nRF52832 with nRF52 SDK 17

My application consists of two timers running every 5 seconds and every 35 ms. After some very long time and only with a few units the timer interrupts do not fire at all. I have a watchdog timer running as well and this does not even get triggered to reset the device. If I reset the device the problem persists Only fix is to completely remove the battery and put it back in. This fixes the issue every time. There are no errors in any of the 'lfclk_request(), app_tmer_init() or app_timer_create()" methods. What could this possibly be? And why can it only be fixed via a hard power cycle?

[Einar Thorsrud]

Hi,

It is very interesting that you have a watchdog enabled, and that also do not reset the nRF. The WDT will force the LF clock on, and it is not possible to turn it off. So the only reason I can think of for this behaviour is that the clock stops counting for some reason, and that it only recovers after a power-cycle. That is not something I have seen before, though.

Do you use a 32.768 kHz crystal, the internal LFRC, or another clock source? Any details would be useful. You write that it only happens on a few units - how many, and out of a how large pool? Can you say roughly how long time it takes before the failure? And are the same devices seem likely to fail again, or do they typically fail only once?

[EduanShuda]

Hi,

Is there some way in which I can test if the LF clock is running?

I use the internal LFRC.

We have about 5 000 units in the field. Do not know how many have this exact problem but it seems like quite a lot. I would say one in every 20 devices.

Also can't say exactly after how long the problem repeats but it is not constant on all devices.

The exact device we are using is the U-BLOX ANNA-B112 if that is of any help.

Another strange thing, when I program a working device none of the timer handlers fire until the watchdog resets the module and then after that everything works fine. If I remove the watchdog timer this does not happen. What in the watchdog timer code would prevent timer handlers to not fire the first time and only fire after a reset by the WDT?

Any help would be appreciated.

[Einar Thorsrud]

Hi,

EduanShuda said:

Is there some way in which I can test if the LF clock is running?

The most straightforward way of doing that is using the RTC to do something. If the COUNTER increases, then the LF clock is running.

EduanShuda said:

when I program a working device none of the timer handlers fire until the watchdog resets the module and then after that everything works fine. If I remove the watchdog timer this does not happen.

Do you mean that if you remove the watchdog then the timer works even from the first reset after programming, or that in this case it never works at all?

EduanShuda said:

What in the watchdog timer code would prevent timer handlers to not fire the first time and only fire after a reset by the WDT?

Nothing should cause that. The WDT is really separate, except that it forces the LF clock on (and you could get an issue where the clock is not routed to the other devices before it is explicitly started, as mentioned).

Can you say something about how you configure and start (and stop if you do that?) the LF clock? Are you using a SoftDevice all the time (which would then start it for you), or not And if you are controlling the LF clock yourself, do you always start the timer when you should, regardless of if it seems to be running or not? I ask because if you check the LFCLKSTAT register (directly or via nrf_clock_lf_is_running()) after a soft reset, the that would indicate that the LF clock is running, but the clock would not be routed to anything other than the WDT. So if you control the clock yourself you should always start it explicitly (nrf_clock_task_trigger(NRF_CLOCK_TASK_LFCLKSTART);) regardless of the state.

[EduanShuda]

Hi,

So I checked the counter of the lfclk with the following: "count = NRF_RTC1->COUNTER"

Upon reset the counter is 0 and stays 0 until the WDT reset occurs then only does it begin counting upwards. If I remove the WDT it counts as expected.

I start the lfclk with the following:

ret_code_t err_code = nrf_drv_clock_init();
APP_ERROR_CHECK(err_code);
nrf_drv_clock_lfclk_request(NULL);
while(nrf_drv_clock_lfclk_is_running());

I then use app_timer_init() and then I create the timers and start them.

I am not using a soft device.

This behavior is really weird and I think it could also have something to do with the fact that after some time the lfclock stops running entirely

[Einar Thorsrud]

Hi,

I agree that this behavior is weird, and there is nothing in your code snippets or description that explain it. But it should not be possible for the LF clock to stop by itself. And if the WDT is in fact running it should not even be possible to stop it if you attempted to (it is forced on in HW when the WDT is enabled). As this is something that should not happen, and you see it on multiple device, I suspect there must be some issue with your firmware. Perhaps the WDT is not always started? And perhaps the clock is disabled inadvertedly at some point under some special circumstances? Can you upload your code so that I can see if I spot any potential issues that could be related to this? You can do that in a private case and refer to this thread if you do not want it to be public.

[Vidar Berg]

EduanShuda, I found this thread via this one: https://devzone.nordicsemi.com/f/nordic-q-a/79488/power_clock_irqhandler-gets-called-after-100seconds-under-unknown-circumstances-causes-the-product-to-freeze. Any chance you may be encountering the clock driver bug I mentioned?