Fast notifications with nRF5340 and nRF Connect SDK

RE: Fast notifications with nRF5340 and nRF Connect SDK

Dumam — Wed, 30 Nov 2022 23:43:35 GMT

Hi,
I've got the same problem as you, but with bt_gatt_write_without_response[_cb]()
nrf5340 is unable to trigger rpmsg(ipc) transfer

Could you tell me how did you changed this call

return k_fifo_get(&free_tx, K_FOREVER);

So it stopped blocking?

RE: Fast notifications with nRF5340 and nRF Connect SDK

Edvin — Thu, 24 Nov 2022 08:36:59 GMT

Hello,

Actually, it points to the same place. I just navigated using VS Code, and there it pointed to ncs\nrf\subsys\bluetooth\rpc\client\bt_rpc_gatt_client.c. I should have checked the .map file.

I believe the reason I couldn't reproduce it was that I just tried to call bt_nus_send() (bt_gatt_notify_cb()) directly from my main() function. I discussed this with a colleague, who recently had a similar case, which I think you should check out:

bt_gatt_notify() can block when running on Zephyr workqueue.

It turns out that it is the workqueue itself locking up. SInce the bt_gatt_notify_cb() is queued by the workqueue, but the workqueue is also required to actually process your BLE queue, you can end up in a deadlock. I guess the takeaway is that you should call bt_gatt_notify from another thread, so where you actually are calling bt_gatt_notify_cb() from now, you should rather trigger a task in another thread, (either the main thread, or a completely separate thread), so that it doesn't end up in a deadlock.

As it is now, this is currently a limitation in the Zephyr Bluetooth Host stack.

Best regards,

Edvin

RE: Fast notifications with nRF5340 and nRF Connect SDK

Benno Trutmann — Wed, 23 Nov 2022 14:42:12 GMT

Hi Edvin.

In this case your stack configuration should be the same as mine. But then it is unclear to my why your code calls the bt_gatt_notify_cb() from bt_rpc_gatt_client.c and not from gatt.c.

Can you verify in zephyr.map from where bt_gatt_notify_cb() is really linked?

In my zephyr.map I see:

.text.bt_gatt_notify_cb
0x000478d0 0xe8 zephyr/subsys/bluetooth/host/libsubsys__bluetooth__host.a(gatt.c.obj)
0x000478d0 bt_gatt_notify_cb

RE: Fast notifications with nRF5340 and nRF Connect SDK

Edvin — Thu, 17 Nov 2022 08:29:24 GMT

For testing, I used the peripheral_uart sample found in ncs\nrf\samples\bluetooth\peripheral_uart.

According to my build folders, CONFIG_BT_RPC is not set. I didn't find any mentions of it in the autoconf.h, but searching for it in my build folder, these were all the hits:

BR,
Edvin

RE: Fast notifications with nRF5340 and nRF Connect SDK

Benno Trutmann — Fri, 11 Nov 2022 13:23:07 GMT

Is it possible that you build your test code with CONFIG_BT_RPC=y "Bluetooth over RPC [EXPERIMENTAL]"?

In this case I see inclusion of code from the "ncs\nrf\subsys\bluetooth\rpc' directory. My project does not set CONFIG_BT_RPC!

RE: Fast notifications with nRF5340 and nRF Connect SDK

Benno Trutmann — Fri, 11 Nov 2022 13:07:14 GMT

Hi Edvin.

To quote the nRF Connect SDK documentation:

"The SoftDevice Controller is implemented by Nordic Semiconductor. The Zephyr Bluetooth LE Controller is an open source Link Layer developed in Zephyr, to which Nordic Semiconductor is a contributor.

Both Link Layers integrate with the Zephyr Bluetooth Host, which completes the full Bluetooth LE protocol stack solution in the nRF Connect SDK. It is possible to select either Bluetooth LE Controller for application development. See Usage in samples for more information."

So with both controllers the Zephyr Bluetooth Host is used, the code for this host is contained in the 'zephyr-ncs\subsys\bluetooth\host' directory. And there bt_gatt_notify_cb() is implemented in gatt.c and calls indirectly conn_tx_alloc() from conn.c in the same directory.

For me it is unclear why you cannot see this call stack in your tests. You mention the function bt_gatt_notify_cb() from "ncs\nrf\subsys\bluetooth\rpc\client\bt_rpc_gatt_client.c", but I don't think this is part of the Zephyr Bluetooth Host.

Regards,

Benno

RE: Fast notifications with nRF5340 and nRF Connect SDK

Edvin — Fri, 11 Nov 2022 09:45:17 GMT

Hello Benno,

Can you please explain what you mean by "configuration with the Zephyr Host and the Softdevice Controller"? How do you set the Zephyr Host when you are using the SoftDevice Controller?

BR,
Edvin

RE: Fast notifications with nRF5340 and nRF Connect SDK

Benno Trutmann — Wed, 09 Nov 2022 12:50:22 GMT

Hi Edvin.

I have no sample code at hand to demonstrate the blocking behavior. At the moment I also have no time to build one.

Are your tests based on a configuration with the Zephyr Host and the Softdevice Controller as in my case?

By the way: I use the version 2.1.1 of the Connect SDK.

Regards,

Benno

RE: Fast notifications with nRF5340 and nRF Connect SDK

Edvin — Mon, 07 Nov 2022 11:23:08 GMT

Hello Benno,

I am still struggling to get the same behavior as you do. Can you please send me a project with the configuration you are using that I can use to reproduce the issue? Are you able to reproduce it e.g. with something based on the peripheral_uart sample?

BR,
Edvin

RE: Fast notifications with nRF5340 and nRF Connect SDK

Benno Trutmann — Wed, 02 Nov 2022 12:02:20 GMT

The controller part was not configured by me, so I do not really know what I am talking about. But I was under the impression that I use the Softdevice controller, but via "HCI using RPMsg" (CONFIG_BT_RPMSG=y).

When I look into the .config file in the app build directory I see no definition of CONFIG_BT_LL_SPLIT nor CONFIG_BT_LL_SOFTDEVICE.

In the .config file of the hci_rpmsg build directory I see

# CONFIG_BT_LL_SW_SPLIT is not set

and

CONFIG_BT_LL_SOFTDEVICE=y

Regards,

Benno

RE: Fast notifications with nRF5340 and nRF Connect SDK

Edvin — Wed, 02 Nov 2022 11:36:16 GMT

Thank you very much Benno.

Are you using the Zephyr Bluetooth Low Energy controller?

Did you set CONFIG_BT_LL_SPLIT=y in your prj.conf (or any other .conf file that is included in your application)?

I believe the reason I struggled to find it is that if you use the Nordic Bluetooth Low Energy controller (SoftDevice controller), which is either set by default, or if you use CONFIG_BT_LL_SOFTDEVICE=y, then the implementation of bt_gatt_notify_cb() that will be used is found in bt_rpc_gatt_client.c (ncs\nrf\subsys\bluetooth\rpc\client\bt_rpc_gatt_client.c), and it looks like this:

int bt_gatt_notify_cb(struct bt_conn *conn,
		      struct bt_gatt_notify_params *params)
{
	struct nrf_rpc_cbor_ctx ctx;
	int result;
	size_t scratchpad_size = 0;
	size_t buffer_size_max = 8;

	buffer_size_max += bt_gatt_notify_params_buf_size(params);

	scratchpad_size += bt_gatt_notify_params_sp_size(params);

	NRF_RPC_CBOR_ALLOC(&bt_rpc_grp, ctx, buffer_size_max);
	ser_encode_uint(&ctx, scratchpad_size);

	bt_rpc_encode_bt_conn(&ctx, conn);
	bt_gatt_notify_params_enc(&ctx, params);

	nrf_rpc_cbor_cmd_no_err(&bt_rpc_grp, BT_GATT_NOTIFY_CB_RPC_CMD,
		&ctx, ser_rsp_decode_i32, &result);

	return result;
}

It doesn't call gatt_notify(). Do you see the same blocking behavior if you try to set CONFIG_BT_LL_SOFTDEVICE=y in your prj.conf?

Best regards,

Edvin

RE: Fast notifications with nRF5340 and nRF Connect SDK

Benno Trutmann — Wed, 02 Nov 2022 09:36:47 GMT

Hi Edvin.

Here is the call stack (keep in mind that conn_tx_alloc() is only called when func in struct bt_gatt_notify_params is set to a callback):

bt_gatt_notify_cb(params->func!=NULL) -> gatt_notify(params->func!=NULL) ->

bt_att_send() -> att_send_process() -> process_queue() -> bt_att_chan_send() -> chan_send() ->

bt_l2cap_send_cb(cb!=NULL) ->

bt_conn_send_cb(cb!=NULL) -> conn_tx_alloc() ->

k_fifo_get(&free_tx, K_FOREVER)

Regards,

Benno

RE: Fast notifications with nRF5340 and nRF Connect SDK

Edvin — Wed, 02 Nov 2022 09:10:59 GMT

Hello, Sorry to bother you with this, but I struggle to find the call path from bt_nus_send_cb() to conn_tx_allloc().

I see that conn_tx_alloc() is used from bt_conn_send_cb(), which again is used from either bt_l2cap_send_cb() or bt_iso_chan_send(). Which one is used in your case? And can you please share the entire callstack from bt_gatt_notify_cb() to conn_tx_alloc()? The reason I am asking is that I need to be able to reproduce this before filing it as a bug report/feature request.

Best regards,

Edvin

RE: Fast notifications with nRF5340 and nRF Connect SDK

Edvin — Mon, 31 Oct 2022 12:03:53 GMT

Hello Benno,

I was not aware that it did that. I agree that it would make sense to have it just return an error instead of waiting for a free buffer. I will report your suggestion (which I agree with) internally.

I guess you can modify the behavior of bt_gatt_notify_cb() if you like, to return -ENOBUFS even if it is not called from System Workqueue context, or you can call it from a different thread that doesn't do anything other than that (so that it doesn't matter if it is blocking).

Best regards,

Edvin

RE: Fast notifications with nRF5340 and nRF Connect SDK

Benno Trutmann — Mon, 31 Oct 2022 09:43:37 GMT

Hello Edvin.

Unfortunately I have no demo code for you to reproduce.

But I think I found the culprit: At the end of the function conn_tx_alloc() in the module subsys/bluetooth/host/conn.c there is the line

return k_fifo_get(&free_tx, K_FOREVER);

And because of that bt_gatt_notify_cb() blocks as soon as the free_tx queue is empty. There is one exception when bt_gatt_notify_cb() gets called from System Workqueue context (e.g. from the "Notification Value callback"), in this case instead of blocking the error code -ENOBUFS is returned.

The size of the free_tx queue can be set by the Kconfig option CONFIG_BT_CONN_TX_MAX, in my tests this option was set to 3.

For many applications this behavior may be the desired one, but for my application a call to bt_gatt_notify_cb() must not block under any circumstances. Therefore I introduced a credit counter in my code, so I can track the fill level of the free_tx queue and only call bt_gatt_notify_cb() if the queue is not empty.

Unfortunately this credit counter is not the real fill level of the free_tx queue and if some other code parts also call bt_gatt_notify_cb(), without respecting the credit counter, my approach could still block.

Therefore I propose a change in the host stack: There is already a struct bt_gatt_notify_params passed to bt_gatt_notify_cb(), it should be easy to add a field to this struct to indicate if bt_gatt_notify_cb() is allowed to block or not.

Regards,

Benno

RE: Fast notifications with nRF5340 and nRF Connect SDK

Edvin — Fri, 28 Oct 2022 12:27:11 GMT

Hello Benno,

Can you please upload a project that I can use to replicate what you are seeing? What are you connected to? Can I reproduce the issue using two DKs of some sort?

Best regards,

Edvin

RE: Fast notifications with nRF5340 and nRF Connect SDK

Caesarn — Thu, 27 Oct 2022 16:30:53 GMT

Great to read your ideas. :))
stumble guys online

RE: Fast notifications with nRF5340 and nRF Connect SDK

Benno Trutmann — Thu, 27 Oct 2022 15:02:06 GMT

Hi all.

For me bt_gatt_notify_cb() does block as well.

I try to implement a flow control with the help of the "Notification Value callback" func in struct bt_gatt_notify_params. But for that to work bt_gatt_notify_cb() should return an error code indicating an out of buffer condition. Whereas in my test bt_gatt_notify_cb() only returns 0.

I use a connection interval of 500ms, the first three calls to bt_gatt_notify_cb() return immediately with return value of 0, but the forth blocks for almost 500ms and then returns also with the value 0.

I too thought that bt_gatt_notify_cb() is non-blocking but the behavior is blocking!

Do I miss some Zephyr BLE config option?

Regards,

Benno

RE: Fast notifications with nRF5340 and nRF Connect SDK

Edvin — Thu, 06 Oct 2022 12:33:41 GMT

So whereabout in the sniffer trace would you say that it starts to slow down?

If you want to actually monitor whether or not the notifications are sent, you should check the return value from bt_gatt_notify_cb. This will tell you whether or not the packet is actually queued.

I guess the reason you are seeing these "slowing down" (they aren't really. You still have on average 1 interrupt every 5ms) is that you have this timer set to interrupt at the same rate as your connection interval (every other timer interrupt). Remember that every 10 ms the nRF needs to spin up the radio, and the softdevice controller takes over to handle the communication with the connected device. Due to clock accuracy (or inaccuracy), these connection events (the events that occur every connection interval) can drift a bit. I believe what you are seeing is that it drifted over to hit the timer interrupts, and in the middle of bt_gatt_notify_cb, the SoftDevice Controller (Bluetooth stack) takes control over the CPU to handle Bluetooth stuff, and then gives back the CPU to your application when it is done. At that point it will continue where it left off.

Best regards,

Edvin

RE: Fast notifications with nRF5340 and nRF Connect SDK

Jason Wright — Wed, 05 Oct 2022 13:48:50 GMT

Hi Edvin,

Sorry, I should have been a bit clearer on this.

Please see L199-201 in main.c for the call to bt_gatt_notify_cb surrounded by P0.04 toggle.
My logic analyzer capture (.sal) and Wireshark capture (.pcapng) are unfortunately not synchronized because I did not have a way of starting them at the same time. When I mentioned 14.5 seconds, I was referring to the logic analyzer capture (see image below).
In the logic analyzer capture, the first notification is sent at 8.694 seconds. The first notification in the Wireshark capture is no. 9472 at 21.044 seconds.
The Wireshark capture includes the connection establishment (starting at no. 7518), MTU exchange, connection parameters, data length extension, etc.

I should also mention that my central device is a nRF52840 dongle running the Nordic connectivity firmware, using the nRF Connect Desktop BLE app, and I can see the notifications appear in the app.

RE: Fast notifications with nRF5340 and nRF Connect SDK

Edvin — Wed, 05 Oct 2022 07:58:18 GMT

Hello,

bt_gatt_notify_cb() is not a blocking call, so it should not be the reason why the P0.04 isn't toggling.

Are you sure that bt_gatt_notify_cb() is even called at 14.5 seconds into the trace? The trace doesn't show any notifications being sent around that time. In fact, it doesn't look like any notifications are sent at all in the connection starting at packet no. 7517 (12.89 seconds into the sniffer trace).

Are you sure bt_gatt_notify_cb() is called at all? And if it is, what does it return? I assume it doesn't return 0, since no notifications are actually sent.

Best regards,

Edvin