app_timer2 timer list corruption seen on rtc overflow

[TomKeddie]

I'm trying to track down an issue where our application gets stuck due to what looks like an infinite loop in the app_timer2 timer list.  When it occurs we see the next pointer of the head element pointing back to the the head.

To detect this condition I've added guards to all the methods in nrf_sortlist.

ASSERT(p_list->p_cb->p_head != p_list->p_cb->p_head->p_next);

I'm seeing this occur repeatedly with the call stack shown above.  on_overflow() calls timer_expire() on a late timer.  Since the timer has a repeat period timer_expire() reschedules the timer but on_overflow() only peeked at the timer, it never removed it from the list.  This timer is still at the head so it gets added to the list in front of itself and the loop is formed.

I'm trying hacks to fix this, I'm fairly sure that on_overflow should be popping the timer from the queue before calling timer_expire()  In the non-overflow case rtc_update() pops the timer from the list before calling rtc_schedule(), which then calls timer_expire().

I'm using an nrf52832 with SDK 15.2 and softdevice 6.1

[TomKeddie]

The issue at https://devzone.nordicsemi.com/f/nordic-q-a/45514/app-timer-list-gets-corrupted-with-sdk-13---dead-lock mentions app_timer but is very similar.

[Einar Thorsrud]

Hi,

I agree with you that it seems very similar to the issue with the old (app_timer.c) implementation mentioned in this thread. It is reported internally, but it has not been resolved yet. The suggested workaround, for now, is to use the app_scheduler.

[TomKeddie]

I saw the mention of app_scheduler, I don't see how the app_scheduler helps runs tasks at a particular time, perhaps you can point me to an example?

Since this only affects repeated timers I've started moving all my repeated timers to use the cc mechanism in an RTC.  There are four cc registers available, this is enough for me for now.

I'd appreciate some investigation into app_timer2 and a resolution in a future sdk.  Thanks.

[Einar Thorsrud]

Hi,

I have bumped the internal ticket so that the SDK R&D team will consider the issue.

[Krzysztof Chruscinski]

Hi,

you are right. On_overflow_evt is definitely missing popping there. I've would change it to something like:

/**
 * @brief Function for handling RTC counter overflow.
 *
 * In case of overflow all active timers must have end value adjusted (reduced to 24 bit range).
 */
static void on_overflow_evt(void)
{
    NRF_LOG_DEBUG("Overflow EVT");
    if (mp_active_timer)
    {
        uint32_t end_val = mp_active_timer->end_val;
        mp_active_timer->end_val = end_val & RTC_COUNTER_COUNTER_Msk;
    }

    app_timer_t *     p_next;
    p_next = sortlist_peek();
    while (p_next)
    {
        if (p_next->end_val <= RTC_COUNTER_COUNTER_Msk)
        {
            p_next = sortlist_pop();
            //If overflow occurs then all timers with value lower than max value expires immediately.
            UNUSED_RETURN_VALUE(timer_expire(p_next));
            p_next = sortlist_peek();
        }
        else
        {
            p_next->end_val &= RTC_COUNTER_COUNTER_Msk;
            p_next = sortlist_next(p_next);
        }
    }
}

I will ensure that it is fixed for next SDK release.