Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

RE: Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

bart — Fri, 26 Nov 2021 08:51:05 GMT

So I tested with 3000us, and it seems like that doesn't result in the loop anymore (I still see plenty of BLOCKED events, but always with 100ms in between, so that's probably the next advertisement with a full time slot).

RE: Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

Mttrinh — Fri, 05 Nov 2021 12:30:37 GMT

[quote user="bart"]Thanks, I will try the first solution and get back with the results. How is this 3800 us calculated, if i may ask?[/quote]

Great! This was based on specification from our Softdevice team, it stated that the mesh stack could always get at least 4000us, even in a connection. It seems to have changed though.

RE: Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

bart — Wed, 03 Nov 2021 10:32:05 GMT

Thanks, I will try the first solution and get back with the results. How is this 3800 us calculated, if i may ask?

Solution 2 is indeed what I also thought to be a better solution, but I don't dare to implement this, as it seems quite a big change, with possible side effects.

RE: Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

Mttrinh — Fri, 29 Oct 2021 14:28:18 GMT

Hi,

From speaking to one of our developer he thinks there are two potential solutions to this:

1. Reduce TIMESLOT_BASE_LENGTH_SHORT_US so it can fit between two connection events of the shortest type. This is the intent of the current functionality, but some condition on the device (configuration, clock drift setting?) is preventing the device from getting this short timeslot. Currently, it's 3800 us, try changing it to 3000, for instance.

2. Add some backoff mechanism in the BLOCKED event handler. In this handler, we're currently setting the timeslot request length to TIMESLOT_BASE_LENGTH_SHORT_US before retrying. If we end up here, and the request timeslot length is TIMESLOT_BASE_LENGTH_SHORT_US already, try starting a backoff timer that comes back to request a timeslot later. This could also be implemented as an nrf_mesh_evt_t, but that might still be too tight of a loop.

RE: Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

bart — Mon, 25 Oct 2021 12:12:04 GMT

Thanks, looking forward to it.

For now I'm using a work around to at least not crash: patch

When just using that work around, however, we get even more calls to sd_radio_request(), causing the main thread to be blocked until the connect succeeded.

To avoid that, I now also wait for NRF_MESH_EVT_DISABLED before calling sd_ble_gap_connect(). This prevents most of the NRF_EVT_RADIO_BLOCKED loops.

Handling NRF_EVT_RADIO_SESSION_CLOSED still takes 4 ms while connecting. And there are still some short NRF_EVT_RADIO_BLOCKED loops, mostly while connected, but they don't block the main thread for seconds, still can take up to 15 ms to handle.

RE: Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

Mttrinh — Thu, 21 Oct 2021 13:36:17 GMT

Hi,

This have been sent to our Mesh team to have look at, I will update you once I have a response from them.

RE: Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

bart — Mon, 18 Oct 2021 13:15:27 GMT

After some more testing, I found out that it seems like sd_radio_request() sometimes is a blocking call, which blocks for p_request->earliest->timeout_us micro seconds, when the SD is connecting as central. Or maybe it's always blocking, and normally returns quickly. After that time, the NRF_EVT_RADIO_BLOCKED event is triggered, and since the main thread is still in the nrf_sdh_soc_evts_poll() loop, it will immediately process this event, leading to another sd_radio_request().

I do not know how to properly fix this, the cpu shouldn't be blocked for that long.

RE: Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

bart — Fri, 15 Oct 2021 14:20:58 GMT

I did some more research, and it's best reproducible with a connection as central.

Before making a connection, I stop the mesh with nrf_mesh_disable(). If i don't stop the mesh, connecting takes very long.

Most times, the mesh actually stops, and i get events NRF_EVT_RADIO_SESSION_IDLE and NRF_EVT_RADIO_SESSION_CLOSED.

Sometimes, however, the mesh doesn't seem to stop, that's when i get multiple NRF_EVT_RADIO_BLOCKED events. What's worse, suddenly the calls to nrf_mesh_on_sd_evt() with the event NRF_EVT_RADIO_BLOCKED take 15ms.

Does the softdevice take up all this CPU time? Or can the call sd_radio_request() take so much time because something in the SD is blocking?

Is there a better way to stop the mesh? It shouldn't be making new timeslot requests when i called nrf_mesh_disable().

RE: Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

bart — Fri, 15 Oct 2021 07:21:03 GMT

Not sure what you mean by connect? Assuming your application are only running ble mesh, there is no concept of connections in a mesh network.

Could you provide more information regarding your use-case and what you are trying to do?

A normal BLE connection with a phone for example, either as peripheral or as central. The mesh is used to communicate between nodes.

Could it be related to interrupt priority levels?

I followed that guide and run all code at thread level: using NRF_SDH_DISPATCH_MODEL_APPSH, and init the mesh with NRF_MESH_IRQ_PRIORITY_THREAD.

Also, is there a reason you are using an old SDK version? If possible I would suggest you move to our latest SDK(nRF5 for Mesh SDK v5.0.0), as many improvements have been made from v3.2.0 to v.5.0.0.

I checked timeslot.c at the 5.0.0 mesh SDK, and that handles the NRF_EVT_RADIO_BLOCKED event in exactly the same way. I didn't have any reason to update SDK version yet, and usually a new SDK requires more resources.

I'm not too sure anymore about NRF_EVT_RADIO_BLOCKED being the source of the problems, I noticed that app_sched_execute() does finish in between the events.

When inspecting the app scheduler queue on crash, I noticed it's full of calls to appsh_events_poll(). A function that gets and dispatches all pending events from the SD. Wouldn't it be a good idea to check whether this call isn't already queued in the scheduler before putting it in there?

RE: Scheduler get full by NRF_EVT_RADIO_BLOCKED events during connect.

Mttrinh — Thu, 14 Oct 2021 14:25:11 GMT

Hi,

[quote user=""]Am I right to assume this timeslot request loop can lead to the scheduler getting filled up (especially when other processes also add to the scheduler)?[/quote]

Yes, I think that can be the case.

[quote user=""]When logging SOC events, I noticed many NRF_EVT_RADIO_BLOCKED events coming in when a connection is made.[/quote]

Not sure what you mean by connect? Assuming your application are only running ble mesh, there is no concept of connections in a mesh network.

Could you provide more information regarding your use-case and what you are trying to do?

Could it be related to interrupt priority levels?

Also, is there a reason you are using an old SDK version? If possible I would suggest you move to our latest SDK(nRF5 for Mesh SDK v5.0.0), as many improvements have been made from v3.2.0 to v.5.0.0.