'POWER_CLOCK_IRQHandler' gets called after ~100seconds under unknown circumstances (causes the product to freeze)

[Mircea Coman]

Hi,

We have a product already on the market for about 2 years, based on nrf52840 and nrf52832. 

The product receives data from an 'external device' via UARTE, forwards the data to a washing machine via UARTE, get response from the machine, sends it back to the device. Repeat.

The device is using a reoccuring freertos timer started with app_timer_create and app_timer_start which updates a TIME variable via a callback. Timer is 'APP_TIMER_MODE_REPEATED'.

Recently we noticed some devices 'hanging' and I found the cause: POWER_CLOCK_IRQHandler' gets called after ~100seconds under, which stops updating the TIME variable via timer callback, and the firmware freezes as it needs time capability.

'nrf_clock_lf_is_running()' returns true and 'NRF_RTC1->COUNTER' is no more incremented.

It seems that if the 'external device' is in the mode which sends lots of data to our device, the bug is occuring. The bug is even occuring when I am not explicitly reading serial data from the external device.

If the device is set in a mode that sends data rarely, the bug does not seem to occur. Also the problem is not occuring if we disconnect the 'external device' to it.

Any ideas what might causing the disabling of the CLOCK interrupt?

Thank you.

[Vidar Berg]

Hello,

Are you enabling the watchdog timer in your application, and in that case, is there a chance that you may have experienced a WD reset? If so, please check if you may be impacted by this bug mentioned here:

https://devzone.nordicsemi.com/f/nordic-q-a/65422/what-are-sdk-17-x-0-known-issues

[Mircea Coman]

We use SEGGER | SDK 12.3.0 | SoftDevice is disabled | WDT is disabled

The bug is reproducible even if SD + WDT are enabled (as they are in the production). This bug went unnoticed for years while tousands of devices were / are running in the field.

Could you tell me why 'POWER_CLOCK_IRQHandler' is called? Looks like device enters power saving mode for some (unknown) reason?

[Vidar Berg]

I see, thanks for confirming. It's hard for me to say what causes POWER_CLOCK_IRQn to be raised and whether it should be expected or not as the trigger source can be any interrupt enabled event in POWER or CLOCK.

Some more follow up questions:

1. Is the problem caused by the program not returning from POWER_CLOCK_IRQHandler(), or does it return, but without the LF clock running?

2. Which interrupts do you have enabled in POWER->INTENSET and CLOCK->INTENSET?  

3. Can you read which event signal have been set when you are in the POWER_CLOCK_IRQHandler?

Thanks,

Vidar

[Mircea Coman]

VIdar, thank you for your answers.

1. 3. Strangely, the event received in POWER_CLOCK_IRQHandler is 'NRF_CLOCK_EVENT_LFCLKSTARTED' , which, by reading the code in 'nrf_drv_clock.c' it should start the LFCLK, instead of stopping it. This is 140 seconds after the device started, and all the while Timer Interrupt was called correctly.

After the call to 'POWER_CLOCK_IRQHandler', the timer callback is not called anymore and 'NRF_RTC1->COUNTER' is no more incremented.

I enabled LOGGING in CLOCK, UART, GPIO but there's no message indicating what is happening.

My recent assumption was that the device was entering low power mode, and I tried setting the mode to 'Constant Latency Mode' by caling 'nrf_power_task_trigger(NRF_POWER_TASK_CONSTLAT);'. Did not help. 

2. The values for POWER and CLOCK (at startup, read in main()):

- NRF_POWER->INTENSET == 2 (?);    

- NRF_CLOCK->INTENSET == 2 (LFCLKSTARTED?)

In 'POWER_CLOCK_IRQHandler'  the values of the 2 registers is zero.

BTW, The clock is initialized in main() by calling 'nrf_drv_clock_init()'

I tried running the code on an development board (instead of the production board) and 'POWER_CLOCK_IRQHandler' is called right at the beginning of device execution...

Ideas?

[Mircea Coman]

What are the conditions under which CLOCK can be stopped intarnally? Is it even possible that the RTC counter is not incredemented anymore, timer callback not called and still, 'nrf_drv_clock_lfclk_is_running()' returns true ? 

[Vidar Berg]

Mircea Coman said:

What are the conditions under which CLOCK can be stopped intarnally?

That should never happen. The application should be in complete control of when the LFCLK is on or off. And most applications tend to keep it always on.

Mircea Coman said:

I tried running the code on an development board (instead of the production board) and 'POWER_CLOCK_IRQHandler' is called right at the beginning of device execution...

Interesting.. So what clock source are you using on your board? Have you tried with a different source? E.g. sdk_config.h::CLOCK_CONFIG_LF_SRC 2 /*Synth */

[Mircea Coman]

Vidar, it worked! Changing Clock source to Synth caused the 'POWER_CLOCK_IRQHandler' to be always called right after device started and the CLOCK did not stop since (wirking as expected).

What I also discovered is that if I enable the SoftDevice by default, the CLOCK also doesn't stop, even if I leave the Clock source to '1' in 'sdk_config.h'.

Thank you!

[Vidar Berg]

Glad to hear that it worked. But please note that the Synthesized clock source is generally not recommend because it may increase the idle current (it forces the HF clock to always be on like UART does).

I'm still not sure why the LFXO is behaving strangely. Have you changed load caps or the 32K crystal recently?

[koniho]

Is it possible that this condition could occur with the softdevice enabled (using HFCLK and external LFCLK)