Fast notifications with nRF5340 and nRF Connect SDK

[Jason Wright]

Board: nRF5340dk

SDK version: v2.1.0

My goal is to transmit 240 bytes of data every 5 ms using GATT notifications and 1 Mb PHY.

In the attached sample, I have my dev kit set up to notify on a characteristic every 5 ms. The connection interval is set to 10 ms, the MTU to 247 bytes, and the data length to 251 bytes. My thought is to fit 240 bytes into a single PDU, and send 2 notification packets per connection interval. I also toggle GPIO P0.04 before and after calling `bt_gatt_notify_cb`.

Looking at a logic analyzer trace of P0.04, I can see that while this setup initially works, eventually `bt_gatt_notify_cb` starts blocking for longer than expected. This happens around 14.5 seconds into the attached capture.

Wireshark capture also attached.

Why is `bt_gatt_notify_cb` blocking this long, and do you have any suggestions to improve the setup?

fast_notifications.zip

[Edvin]

Hello Benno,

Can you please upload a project that I can use to replicate what you are seeing? What are you connected to? Can I reproduce the issue using two DKs of some sort?

Best regards,

Edvin

[Benno Trutmann]

Hello Edvin.

Unfortunately I have no demo code for you to reproduce.

But I think I found the culprit: At the end of the  function conn_tx_alloc() in the module subsys/bluetooth/host/conn.c there is the line

return k_fifo_get(&free_tx, K_FOREVER);

And because of that bt_gatt_notify_cb() blocks as soon as the free_tx queue is empty. There is one exception when bt_gatt_notify_cb()  gets called from System Workqueue context (e.g. from the "Notification Value callback"), in this case instead of blocking the error code -ENOBUFS is returned.

The size of the free_tx queue can be set by the Kconfig option CONFIG_BT_CONN_TX_MAX, in my tests this option was set to 3.

For many applications this behavior may be the desired one, but for my application a call to bt_gatt_notify_cb() must not block under any circumstances. Therefore I introduced a credit counter in my code, so I can track the fill level of the free_tx queue and only call bt_gatt_notify_cb() if the queue is not empty.

Unfortunately this credit counter is not the real fill level of the free_tx queue and if some other code parts also call bt_gatt_notify_cb(), without respecting the credit counter, my approach could still block.

Therefore I propose a change in the host stack: There is already a struct bt_gatt_notify_params passed to bt_gatt_notify_cb(), it should be easy to add a field to this struct to indicate if bt_gatt_notify_cb() is allowed to block or not.

Regards,

Benno

[Edvin]

Hello Benno,

I was not aware that it did that. I agree that it would make sense to have it just return an error instead of waiting for a free buffer. I will report your suggestion (which I agree with) internally. 

I guess you can modify the behavior of bt_gatt_notify_cb() if you like, to return -ENOBUFS even if it is not called from System Workqueue context, or you can call it from a different thread that doesn't do anything other than that (so that it doesn't matter if it is blocking).

Best regards,

Edvin

[Edvin]

Hello, Sorry to bother you with this, but I struggle to find the call path from bt_nus_send_cb() to conn_tx_allloc().

I see that conn_tx_alloc() is used from bt_conn_send_cb(), which again is used from either bt_l2cap_send_cb() or bt_iso_chan_send(). Which one is used in your case? And can you please share the entire callstack from bt_gatt_notify_cb() to conn_tx_alloc()? The reason I am asking is that I need to be able to reproduce this before filing it as a bug report/feature request.

Best regards,

Edvin

[Benno Trutmann]

Hi Edvin.

Here is the call stack (keep in mind that conn_tx_alloc() is only called when func in struct bt_gatt_notify_params is set to a callback):

bt_gatt_notify_cb(params->func!=NULL) -> gatt_notify(params->func!=NULL) ->

bt_att_send() -> att_send_process() -> process_queue() -> bt_att_chan_send() -> chan_send() ->

bt_l2cap_send_cb(cb!=NULL) ->

bt_conn_send_cb(cb!=NULL) -> conn_tx_alloc() ->

k_fifo_get(&free_tx, K_FOREVER)

Regards,

Benno