NRF54L15 Timer interrupt not generating after few cycles

[darav]

We are using nRF54L15 with Zephyr + (NCS) for a low-power application.

Our firmware performs the following cycle repeatedly:

1. Wake up from System OFF

2. Perform data acquisition and program flow

3. Configure a GRTC wakeup interrupt for the next wake-up period

4. Enter System OFF again

The wake-up timing is controlled through GRTC interrupts.

Observed Issue

The system runs correctly for the initial cycles (typically for ~3 days of operation). After this, we observe that one of the timers fails — the GRTC interrupt is generated, so the device wakes up, but one of the main timers our application uses does not generate interrupts which is an important base for our application resulting in application halt.

Our usage -:

K_TIMER_DEFINE(APP_timer, Timer_handler, NULL);  // define the timer

void init_APP_timer(void)

{

    k_timer_start(&APP_timer, K_MSEC(SYSTICK_INTERVAL_MS), K_MSEC(SYSTICK_INTERVAL_MS));

}

void sysTick_handler(struct k_timer * timer)

{

}

While investigating, we found that this behavior appears related to the following erratum:

• nRF54L15 Rev2 anomaly L15_25
  https://docs.nordicsemi.com/bundle/errata_nRF54L15_Rev2/page/ERR/nRF54L15/Rev2/latest/anomaly_L15_25.html

The errata describes a scenario where the compare event may fail to trigger, and it lists several workarounds.

We would appreciate guidance on the following :

1. Implementation in Zephyr/NCS:
   The errata describes the workaround in terms of bare-metal handling. What is the recommended way to implement this workaround when using Zephyr + NCS, where Timers are typically managed by zephyr apis?

2. Reliability after applying workaround:
   Once the workaround is implemented, is the GRTC interrupt mechanism considered fully reliable for long-term periodic wake-up from System OFF, or are there any additional precautions recommended?

Any clarification on the correct approach for implementing the errata workaround in a Zephyr-based firmware would be greatly appreciated.

Thanks.

[Vidar Berg]

Hello,

Are you using the z_nrf_grtc_wakeup_prepare() function as in the system OFF sample here: https://github.com/nrfconnect/sdk-zephyr/blob/main/samples/boards/nordic/system_off/src/main.c before entering System OFF? Also, which SDK version are you on?

While investigating, we found that this behavior appears related to the following erratum:

• nRF54L15 Rev2 anomaly L15_25
  https://docs.nordicsemi.com/bundle/errata_nRF54L15_Rev2/page/ERR/nRF54L15/Rev2/latest/anomaly_L15_25.html

This applies to the TIMER peripheral, but the zephyr timer is using GRTC.

Best regards,

Vidar

[darav]

Yes, we are using z_nrf_grtc_wakeup_prepare() function as in the system OFF sample, immediately before entering system OFF.
We are using SDK version 3.0.2

Vidar Berg said:

This applies to the TIMER peripheral, but the zephyr timer is using GRTC.

Oh. Then what can be the reason for the behaviour we are observing ?

[Vidar Berg]

Thanks for confirming. We will need to troubleshoot this further to understand why the timer instance is failing to run, as there is currently not much information to go on. All Zephyr timer instances and the scheduler run off the GRTC, so if there were an issue with the GRTC triggering, I would expect all timer instances to fail, not just one. Is it always the same timer that fails to run? 

I suggest modifying the project to use shorter sleep intervals (or other adjustments) to see if you can reproduce the issue more frequently, which should make debugging easier.

[darav]

Vidar Berg said:

so if there were an issue with the GRTC triggering, I would expect all timer instances to fail, not just one. Is it always the same timer that fails to run? 

I dont think there is an issue with GRTC triggering, as the device does wake up at stipulated interval via GRTC interrupt from system off mode, after which the timer instance fails.That is the only timer instance we are using based on which other timing related activities are executed.

Sure, we will reduce the sleep intervals and check if we can reproduce it more frequently and taper down the scenario variables. Do you have any suggestions for some debug prints that we might need to debug this better?

[Vidar Berg]

Yes, but the timer instance is also running off the GRTC. 

so the device wakes up, but one of the main timers our application uses does not generate interrupts

darav said:

.That is the only timer instance we are using based on which other timing related activities are executed.

I'm not sure how to interpret this. My understanding is was that you were starting several timers, but that only one was failing.

[darav]

Vidar Berg said:

.That is the only timer instance we are using based on which other timing related activities are executed.

I'm not sure how to interpret this. My understanding is was that you were starting several timers, but that only one was failing.

sorry for the unclear response. we are using that single timer instance. 
For debugging purposes what we implemented following code -: 

/* ticks variable is incremented from the timer instance ISR at frequency of once every milisecond*/
while(1){
    main_application();
    if(ticks>1){
        printf("some debug print");
        task_wdt_feed(task_wdt_id);
    }
}

Due to this implementation logically, if the timer is working properly ticks variable will be greater than 1 and the if condition will be true which will keep on feeding watchdog.
When the timer instance fails, the if condition is not satisfied for period greater than watchdog timeout and the device should reboot from watchdog elapsed callback.

But now what we've observed is when the issue for timer instance is reproduced, the if condition is not satisfied and yet the watchdog timer callback is not called and device is not rebooted. so i'm guessing that when the timer instance is failing, watchdog is also failing.

One more observation - If we keep the device wakeup intervals to be short (every ~2 mins) the issue does not seem to be reproduced. But occurs almost everytime when the wakeup interval is >30 mins, after 18 hours or sometimes more.

[Vidar Berg]

Thanks for confirming. 

darav said:

Due to this implementation logically, if the timer is working properly ticks variable will be greater than 1 and the if condition will be true which will keep on feeding watchdog.

Do you have logs or something that confirms that k_timer_start() actually got called and that it was not something else that failed during startup? Please also try to test your minimal application with SDK v3.2.4 (latest stable release at time of writing)

[darav]

yes, we do have a log static that "k_timer()" is actually called which is observed even when the issue is produced.
Sure, i'll try my application with v3.2.4 and update you on it

I think this is the issue/bug we are observing -: 
https://github.com/zephyrproject-rtos/zephyr/pull/91432
github.com/.../89147

[Vidar Berg]

Thank you for the update. I remember the issue you linked now, and I agree that it is likely be the cause of the problem you have experienced.