App_timer expires immediately - SDK 10

[Nikolaus Wittenstein]

I am seeing a problem identical to what's described in this question. I'm using SDK version 10.

My timers work fine for a while, but then suddenly and seemingly at random all timers expire the instant they are started. Sometimes it goes away and the timers work again, but sometimes the only fix seems to be a restart.

I've seen two suggested solutions. One is to add a delay or a call to rtc1_compare0_set in compare_reg_update; I'm on SDK 10 and both solutions are already there:

    ...
    rtc1_compare0_set(rtc1_counter_get());  // this should prevent CC to fire again in the background while the code is in RTC-ISR
    nrf_delay_us(MAX_RTC_TASKS_DELAY);
    timer_timeouts_check_sched();
    ...

The only other solution I've found on the forum is to have a dummy timer that is always running. It's recommended in this answer.

Sadly, I do already have a dummy timer always running—I've been using it to force the RTC to stay on so I can use it for timekeeping.

Are there any other known problems or solutions with app_timer? I'm running out of ideas. Thanks!

[Nikolaus Wittenstein]

We only have one board available right now, unfortunately, so I can't really send it out.

In the mean time I've been looking into things myself. I'm printing out (using RTT) the counter value from rtc1_counter_get inside of timer_timeouts_check. Here's a mysterious progression:

   ...
    0x0DEBA9
    0x0E5B26
    0x0E6F14
    0x8E6F5E
    0x8E6FCE
    ...

Somehow the high bit in the counter is getting set. This coincides exactly with when all timers begin to expire instantly (note the very close tick count gaps at the end there). Is there any possible way the RTC1 counter could get screwed up like this? I'm not using the counter at all except through app_timer.

I'll comment again when I have further results.

Thanks!

[Nikolaus Wittenstein]

When the RTC jumps like that and a timer gets scheduled, the timer's ticks_to_expire gets set to 0 in list_insertions_handler because the counter is so far ahead of m_ticks_latest.

Then, in timer_timeouts_check, ticks_expired is 0 because it gets set to the timer's ticks_to_expire. m_ticks_elapsed consequently gets set to 0, so m_ticks_latest never increases. At this point, any timer that gets scheduled will expire immediately.

A solution that appears to work is the addition of the following lines of code:

+        // If we processed all timers, then move the elapsed ticks counter all the way up to now.
+        if (p_timer == NULL)
+        {
+            ticks_expired += ticks_elapsed;
+        }
         // Queue the ticks expired.
         m_ticks_elapsed[m_ticks_elapsed_q_write_ind] = ticks_expired;

(see next comment)

[Nikolaus Wittenstein]

Here I'm checking to see if p_timer is NULL, which signifies that we've processed the whole list of timers. If we've done so, we add ticks_elapsed to ticks_expired so that the new total ticks_expired will be equal to the original ticks_elapsed before we started processing timers. Consequently, if all the timers in the list have a ticks_to_expire of 0, we'll still move m_ticks_elapsed forward. I think this should work as a general solution though I haven't put it through any sort rigorous testing. I haven't been able to reproduce the instant-expiration problem since adding it.

The question then becomes: why does the RTC jump like that? Any ideas?

[Nikolaus Wittenstein]

I think there may also be something incorrect happening in list_insertions_handler. If no timer has expired for more than half of the RTC duration then m_ticks_latest won't have been updated during this time. list_insertions_handler will fail this check:

    if (
         ((p_timer->ticks_at_start - m_ticks_latest) & MAX_RTC_COUNTER_VAL)
         <
         (MAX_RTC_COUNTER_VAL / 2)
        )

delta_current_start will be set to a very large value, and the timer's ticks_to_expire will get set to 0.

This means that if you have a timer running with a timeout greater than MAX_RTC_COUNTER_VAL/2 and don't expire any timers for just over MAX_RTC_COUNTER_VAL/2, then setting a timer with a reasonable timeout will cause it to expire immediately.

This is also relevant because it doesn't work with my patch above. See next comment.

[Nikolaus Wittenstein]

My first patch assumed that we would only ever see ticks_to_expire == 0 when there was only one timer in the list. I had changed app_timer very slightly to keep RTC1 on at all times, so it was possible to be inserting a timer into an empty list while RTC1 was running.

For my latest comment I thought it prudent to roll back to stock app_timer and lock the timer on using a timer with a very long timeout to make sure that I could repro that instant-expire bug without my changes. Imagine my surprise when the instant-expiration bug reappeared. The problem is obvious: I was checking if we'd processed the whole list, which of course we hadn't because the second timer was so long.