FreeRTOS and SoftDevice events: hard fault

[Pertti Kasanen]

We have used FreeRTOS a lot, also on nRF51 and nRF52. Now I'm trying to migrate our nRF52 code form SDK11-alpha and SoftDevice 2.0.0-7-alpha (?) to the most recent versions. Most of things are working but one problem remains. We use s132_nrf52_2.0.1_softdevice.hex.

Everything runs fine for a few minutes, but then some race condition occurs and causes a hard fault. Possibly when a lower priority task is preempted, due to a tasks invoked by calling intern_softdevice_events_execute();

Question 1) Any ideas why in the SoftDevice event handler (ble_new_event_handler() copied straight form the SDK11 example ble_app_hrs_freeertos) fails with a stack trace like this:

HardFault_handler () ... gcc_startup_nrf52.s
<signal_handler called>
0x000092a in ?? ()
<signal_handler called>
... in vPortStartFirstTask()

Otherwise this looks good to me but should have exited the previous signal handler before the new one gets called. This stack trace is (manually) copied probably from a test running the signal handled event task on the lowest FreeRTOS priority.

If using a high FreeRTOS task priority, then the same error happens at portYIELD_FROM_ISR(yield_req) in the ble_new_event_handler(). Specificly the hard fault happens in portmacro_cmsis.h at the end of the portYIELD() call at the __ISB(); line.

Our application logic can manage with omitting the portYIELD_FROM_ISR() call in ble_new_event_handler() and that seems to fix the instability. This is not optimal.

Question 2) Is there somewhere documentation on how the SoftDevice handles mutual exclusivity - in RTOS context? We use several FreeRTOS tasks and priorities, with multiple of them making calls to softdevice functions...

Is there a mutex on the SoftDevice side so that we can just call the functions without putting a mutex sempahore around every SD call (which would become cumbersome with asynchronous calls). And is there something we should avoid when processing events firm SoftDevice and making new softdevice calls based on those?

I'm wondering it some of these issues is causing the trouble above...

[RK]

+1 for qu. 2 which has been bothering me. If the RTOS task runs at lower priority than the SVC, everything should be fine, but if it runs at a higher priority sooner or later, probably sooner, it's going to interrupt something in a SVC call, stack the current state, inside the SD then task switch to something else. If that then also does an SVC call I can't think it would end well.

I wouldn't expect the SD to have any internal mutexes as it doesn't really need them , anything which can interrupt an SVC call can't call another one (can't SVC from a higher priority interrupt), anything of lower priority won't run, so there's an automatic protection.

With an RTOS which can interrupt an SVC call and then adjust the stack to return to somewhere totally different, you lose that. I believe the RTOS port however has the scheduler running at higher priority than SVC, don't see how that works.

[Pertti Kasanen]

Thanks RK for good speculation.

I'm wondering for example: should I put the soft device event handler task under/above/same FreeRTOS level as the TmrS task of the FreeRTOS port.

Reading this infocenter.nordicsemi.com/index.jsp looks like we should try to use only interrupt levels 6 and 7 with RTOS application level interrupts - or otherwise be very specific what to start from those higher level interrupts.

I recall having somewhere interrupt level 3 in use - will check that next...

[RK]

They're just my musings at this point- I'm still thinking it through and need to spend some time doing a FreeRTOS deep dive until I really feel comfortable with it.

I just looked at a few more of the examples and actually it seems the kernel priority is 0x0f, the lowest level (and this port doesn't seem to follow the documented convention that it should be the fully-shifted, lowest bits set to 1 version, ie 0xff). The highest available interrupt which can use FreeRTOS ISR calls is set to 1, which is lower than the softdevice critical priority, which makes sense. With that the case, you shouldn't have to protect SVC calls as the kernel can never interrupt one and task switch you. That would be the case with the kernel interrupt level anywhere between 0x4 and 0xf.

interrupt priority 3 should be fine, you can even call RTOS functions from it. So your actual problem must lie elsewhere.

[Pertti Kasanen]

Edited the link to infocenter.nordicsemi.com/index.jsp , that was the page ment.

[Pertti Kasanen]

Edit: The problem was configLIBRARY_LOWEST_INTERRUPT_PRIORITY. Got it working earlier, but took this long :( to do all the testing and removing my workarounds etc. I did "check" that priority setting already earlier, but due to mixed up #ifdef:s the line with 0x0f was not in use...

That fixed the all the issues with portYIELD() and ble_new_event_handler()

---

I may have found part of the problem. Every couple of minutes there used to be hard faults with stack trace like this.

It seems to me that there is a flaw on the FreeRTOS port and it's integration to SoftDevice event handling. It (probably only) appears when ble_new_event_handler causes a FreeRTOS task to yield AND there are multiple events in the soft device queue.

The ble_new_event_handler (as copied from the SDK examples) enables soft device interrupts when yielding with portYIELD(). But it should not - before all the queued events have been processed in intern_softdevice_events_execute() in softdevice_handler.c

Wrote a workaround and this error with SWI2_EGU2_IRQHandler has not appeared since. I think there is no way doing this without modifying the intern_softdevice_events_execute() code. Need to do some more experimenting, but I'll open a support ticket on this.