Device doesn't wake up from low power mode on Zephyr timer events

[Alex_S@Aperia]

Our board is based on nRF52833, using SDK 2.1.2, Zephyr 3.1.99

Project is configured to enter low power mode for both CPU and peripherals

CONFIG_PM=y
CONFIG_PM_DEVICE=y

Our Zephyr application contains a timer that fires up every 5 min and performs certain activity (including advertising BLE)
I'd expect that timer event (interrupt) would wake device up from low power mode 
Unfortunately sometimes it doesn't.
Occasionally we discover devices that are "frozen" and stay in low power mode until GPIO interrupt, which ALWAYS causes device to wake up.
The problem is that I can't rely on this GPIO interrupt to happen periodically enough, timer is my last line of defense.
When I set breakpoint int he debugger, after waking up, I'm always at next instruction after WFI (low power mode Wait for interrupt)

I'm wondering if timer interrupt is not configured properly in the device tree somewhere...

[Einar Thorsrud]

Hi,

Can you explain/show in detail how you have configured the timer interrupt?

[Alex_S@Aperia]

Hi Einar,
I didn't explicitly configure timer interrupt, you can see that in my attached device tree files, so it is whatever configuration is the default chosen by SDK (again, I'm at 2.1.2)
I also attached device tree files created by build for reference

Thank you,
Alex

[Alex_S@Aperia]

>What is it that should wake the system up, then? 
I would expect the system to be waken up by one of the timer interrupts configured by SDK during build. My board device tree files don't explicitly configure timer (see 32-00008618.dts attached) so I assume that build process uses defaults, because as you can see in ./build/zephyr/zephyr.dts, timer interrupts are configured (and are working properly 99% of the time)

t

imer0: timer@40008000 {
compatible = "nordic,nrf-timer";
status = "okay";
reg = < 0x40008000 0x1000 >;
cc-num = < 0x4 >;
interrupts = < 0x8 0x1 >;
prescaler = < 0x0 >;
};
timer1: timer@40009000 {
compatible = "nordic,nrf-timer";
status = "okay";
reg = < 0x40009000 0x1000 >;
cc-num = < 0x4 >;
interrupts = < 0x9 0x1 >;
prescaler = < 0x0 >;
};
timer2: timer@4000a000 {
compatible = "nordic,nrf-timer";
status = "okay";
reg = < 0x4000a000 0x1000 >;
cc-num = < 0x4 >;
interrupts = < 0xa 0x1 >;
prescaler = < 0x0 >;
phandle = < 0x9 >;
};

>Please be specific, and if you can refer to code that would also be good. 
This is initialization of timer and workqueue

k_timer_init(&park_state_timer_id, park_state_timeout_handler, NULL);
k_work_init(&park_state_work_id, park_state_timeout_work_handler);

This is timer's handler

void park_state_timeout_handler(struct k_timer *timer)
{
    k_work_submit(&park_state_work_id);
    return;
}

This is work queue handler 

static void park_state_timeout_work_handler(struct k_work *work)
{
    start_ble_advertisement();

    return;
}

>Without it, I am unfortunately not able to understand the issue or suggest ways to resolve it.
Let me try to rephrase the problem statement:
1. My system spends most of the time in Idle Sleep mode - Zephyr threads have no work to do, Idle thread takes over and enters sleep mode
2. It is designed to be awakened either by interrupts from physical pins or by aforementioned periodic timer interrupts
3. That awakening is working fine 99.9% of the time
4. On rare occasions I find that timer handler stops firing and system remains in Idle Sleep
5. When in this state - GPIO interrupt always wakes it up

Please let me know if there is any specific information or piece of code I can provide.

[Einar Thorsrud]

Hi,

That is not expected. In practice, virtually all applitactions work like this waking up from time to time to perform certain tasks. Zephyr use the RTC for this, which runs of the 32.768 kHz clock (not a timer, which runs of the HF clock and is too power consuming). I do not recall seeing an issue like this before and given that this approch is virtuall universal, I wonder if you have tested multiple devices? Do you see this on all, or just one?

If this issue is only reproducabel on one of many devices, we can look into if there could be an hardware failure. If not, it would be good to know more about your application. Could there ber another reason it does not behave as it should? For instance, could it be an issue related to qork queue or similar? Perhaps it is full due to some issue/bug causing problems processing work? Or somethign else? Have you done more debugging to see the state of the device when it does not wake up on time?

[Alex_S@Aperia]

Hi Einar, 
You are absolutely correct, it is RTC and not a timer, my board has RTC running of 32 KHz crystal,
and I have CONFIG_NRF_RTC_TIMER=y
The problem has so far been reproduced on 20 or so different boards (out of thousands), so its not a one off. It is not reproducible at will unfortunately.
On one occasion I was able to attach debugger to board in this state, and found myself at break point on next instruction after WFE (I suspect connecting JTAG generated an interrupt). Unfortunately I didn't do any further debugging at that time, wasn't sure what to look at. Once the PC was past WFE - everything went back to normal and Zephyr started running
It is of course possible that the problem is caused by a bug in app, full work-queue and such...

[Einar Thorsrud]

Hi,

I see. Do you have the hardware watchdog enabled? If not, it would be intreresting to see if using the watchdog would help, so that the device would reset in case it is in this bad state. One reason is that it could be a potential fail-safe/workaround if the issue is not fixed, but also as the watchdog is based on the LFCLK, and if the watchdog functions, that means that the LFCLK is also running. Given that this issue is difficult to reproduce I am not sure how easy it is to test, though?

Focusing on the firmware side of things, could there be a bug in your fimrware that could create a situation were there are no threads that have any work to do at any time in the future withotu a GPIO interrupt? So that other than the idle task which enters WFE there is no work to be done before a GPIO interrupt happens?

[Alex_S@Aperia]

Hi Einar,
Thank you, HW watchdog is a great idea, I was thinking I need to implement it regardless of this problem.
There is a bit of concern that If max WD timer is not long enough - and therefore needs to be pet frequently - it could have impact on my battery life, but I suspect it would be tolerable.
Could you please point me to concise example of WDT implementation ?
The only example I found in SDK 2.1.2 is MemFault demo, and its very memfault-specific
Thank you,
Alex

[Einar Thorsrud]

Hi Alex,

I suggest you refer to the watchdog driver sample under zephyr/samples/drivers/watchdog/ (this link is for SDK v2.9, but the sample exist in 2.1.2).