FreeRTOS Race Condition with Tickless Idle and Suspending Tasks

RE: FreeRTOS Race Condition with Tickless Idle and Suspending Tasks

Susheel Nuguru — Wed, 20 Jan 2021 10:07:13 GMT

Without touching all the application ISRs, I think it is quite difficult to solve this corner case.

RE: FreeRTOS Race Condition with Tickless Idle and Suspending Tasks

A Knecht — Mon, 18 Jan 2021 16:28:44 GMT

Generally I agree. Modifying all the interrupt handlers to modify an additional atomic variable is kind of a last resort solution, but probably the one I'll have to use.
But I'm not yet sure that it is feasible to modify all the interrupt handlers. Some might be hidden in binary 3rd party code that I am unable to modify and some might be in Nordic SDK code, which I can change, but would like to do so as little as possible.

RE: FreeRTOS Race Condition with Tickless Idle and Suspending Tasks

Susheel Nuguru — Tue, 12 Jan 2021 14:59:35 GMT

[quote user="aknecht"]Say I have an interrupt that sets some atomic variable that leads the main loop to do something the next time it runs and given the following sequence switching away from the main task:[/quote]

Can't you include the atomic variable into your if condition? for example

before you suspend your main task, clear the atomic variable and the vTaskSuspend(main_task)
in your interrupt service handler, set the atomic variable

in configPOST_SLEEP_PROCESSING

if( (NVIC->ISPR[0] | NVIC->ISPR[1] | NVIC->ISPR[2])  | (atomic_variable == 1))
{
    vTaskResume(m_main_task_handle);
    atomic_variable = 0;
}

I think the only possibility that the atomic_variable in that IF statement gets through is when the corner case you mentioned happens.

Do you agree?

RE: FreeRTOS Race Condition with Tickless Idle and Suspending Tasks

A Knecht — Mon, 11 Jan 2021 11:02:21 GMT

It's not about an observable behavior, but about theoretical correctness.

- sd_app_evt_wait gets called inside a critical region. Hence interrupts that wake the CPU and resume code execution after sd_app_evt_wait will be pended and their presence can be checked via the NVIC->ISPR flags.
- sd_app_evt_wait will not sleep if an interrupt has occurred before it was last called, according to the note in the doxygen comment

If an application interrupt has happened since the last time sd_app_evt_wait was

* called this function will return immediately and not go to sleep. This is to avoid race

* conditions that can occur when a flag is updated in the interrupt handler and processed * in the main loop.

- Due to the way FreeRTOS is written, however, reaching this code after running some other task implies that an interrupt has occurred because task switching is implemented using software interrupts. Hence, sd_app_evt_wait will not sleep.
- If no task is executable after running vPortSuppressTicksAndSleep, it will get called again from the loop in prvIdleTask. This time it will sleep if no interrupt has occurred.

Hence, if I do not check the NVIC->ISPR flags in my configPOST_SLEEP_PROCESSING handler and simply resume the main task, the system will never sleep since every call to sd_app_evt_wait is preceded by an interrupt (the one switching to the idle task).
If I do check the NVIC->ISPR flags in my configPOST_SLEEP_PROCESSING I might sleep too much due to a very similar race condition that the non-FreeRTOS use of sd_app_evt_wait is free of.

Say I have an interrupt that sets some atomic variable that leads the main loop to do something the next time it runs and given the following sequence switching away from the main task:

1 vTaskSuspend(main task)
2 switch over to idle task via software interrupt
3 prvIdleTask calls portSUPPRESS_TICKS_AND_SLEEP
4 the critical region is entered
5 sd_app_evt_wait is called, returns immediately
6 configPOST_SLEEP_PROCESSING is called
7 configPOST_SLEEP_PROCESSING checks the NVIC->ISPR flags and decides the main task must not yet be unsuspended
8 the critical region is exited
9 prvIdleTask calls portSUPPRESS_TICKS_AND_SLEEP
10 the critical region is entered
11 sd_app_evt_wait is called, this time it sleeps
12 ...

If the interrupt that sets the atomic variable happens anytime after 1 and before 4, the main task may not get unsuspended until much later by another interrupt. It cannot be checked in the NVIC->ISPR flag because it happened before entering the critical region and it will not cause the second sd_app_evt_wait to return immediatley (it will do so for the first sd_app_evt_wait, but the first one returns immediately in all cases because of the software-interrupt based task switching).

Are these correct assumptions?
Let me know if this helps or if I should attempt some minimal code example outlining my concern.

RE: FreeRTOS Race Condition with Tickless Idle and Suspending Tasks

Susheel Nuguru — Mon, 11 Jan 2021 08:48:23 GMT

Thanks for clearing that. Can you please do the same test with FreeRTOS from SDK17.2 and see if you see the same issue. If so, then I will have to spend more time on this.

RE: FreeRTOS Race Condition with Tickless Idle and Suspending Tasks

A Knecht — Sun, 10 Jan 2021 14:57:11 GMT

Thaks for the tip concerning SDK 17.2. I'll copy over the new code.
Concerning configPOST_SLEEP_PROCESSING I'm not calling it manually at all. It is called from vPortSuppressTicksAndSleep defined in port_cmsis_systick.c. The only thing I'm calling is vTaskSuspend for the main task (enabling the idle task to run) and vTaskResume from within configPOST_SLEEP_PROCESSING as in the snippet in the original post.

Thanks

RE: FreeRTOS Race Condition with Tickless Idle and Suspending Tasks

Susheel Nuguru — Mon, 04 Jan 2021 10:47:16 GMT

Hi Andreas,

There were few bug fixes that came into the SDK after SDK14, I recommend you to download SDK17.2 and replace your old SDK14\external\freertos folder with the new SDK17.2\external\freertos folder. This should work fine as this folder is independent of other SDK modules.

Regarding the race condition,

Where are you calling configPOST_SLEEP_PROCESSING from? if this is in the interrupt context then you should use xTaskResumeFromISR inside this macro.
It seems that your suspicion is correct about the race condition in the way you resume the main task, but i still do not have a full picture as i need to see the code where you call configPOST_SLEEP_PROCESSING. Can you please attach the code snippet where you used it so that i can get better understanding of the probable race condition.

RE: FreeRTOS Race Condition with Tickless Idle and Suspending Tasks

Marte Myrvold — Wed, 23 Dec 2020 11:48:37 GMT

Hi,

I'm afraid our FreeRTOS expert is out of office because of holiday season, and will not be back before the new year. I'll forward your case to him as soon as he's back.

Best regards,

Marte