BLE multilink ATT timeout problem occurs when the number of connections increases

[hmw]

Recently I am using BLE in Zephyr to implement the multilink central function
However, none of the references currently have a similar function, and there is also the nordic zephyr code
https://github.com/nrfconnect/sdk-nrf  <-There are only examples of multilink peripherals in this link
So I wrote one to implement it. The goal I want to achieve is one central to 30 peripherals
The current connection with 4 peripherals is fully stable, but when I increase the connected peripherals, the following error will appear

<err> bt_attLATT Timeout
<wrn> bt_att: No ATT channel for MTU 5
<wrn> bt_att: No pending ATT request

The picture below shows the error when connecting 20 peripherals

At present, I know that modifying the Interval connection will indeed improve, but only to make <err> bt_attLATT Timeout happen later.

How can I avoid this problem so that I can connect 30 peripherals stably?

Below is my code
Or you can go to this page to download https://github.com/mfinmuch/zephyr-ble-mulrilink-test

3225.multilink central.rar

Thanks,

Poyi

[Hung Bui]

Hi Hmw, 

Could you describe a little bit more about your application ? 
What's the connection interval you used ? What data do you send/receive , which direction ? 

You actually got a MPU fault, suggesting there could be a stack overflow. Please try increasing this: CONFIG_BT_RX_STACK_SIZE. Try double the stack size, maybe to 4096. 
Also, try to increase these if it's already as follow: 

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_MAIN_STACK_SIZE=2048

Are you using Zephyr stack or Nordic stack ?

[hmw]

Currently I set the interval as follows

#define MIN_CONNECTION_INTERVAL         72     
#define MAX_CONNECTION_INTERVAL         72    
#define SLAVE_LATENCY                   0    
#define SUPERVISION_TIMEOUT             50 

I use the BLE example central_hr and peripheral provided by Zephyr to make changes, it should be Zephyr stack

My central will look for peripherals first
Establish a connection, peripherals will send data to central(use notify), and central will return data to it (use bt_gatt_write)

And the data is just a simple sequence

I thought only the following parameters need to be set

CONFIG_HEAP_MEM_POOL_SIZE=4096
CONFIG_BT_RX_BUF_LEN=258
CONFIG_BT_ATT_TX_MAX=10
CONFIG_BT_ATT_PREPARE_COUNT=10
CONFIG_BT_CONN_TX_MAX=18
CONFIG_BT_L2CAP_TX_BUF_COUNT=18
CONFIG_BT_L2CAP_TX_MTU=247
CONFIG_BT_L2CAP_RX_MTU=247
CONFIG_BT_L2CAP_DYNAMIC_CHANNEL=y
CONFIG_BT_CTLR_PHY_2M=y
CONFIG_BT_CTLR_RX_BUFFERS=18
CONFIG_BT_CTLR_TX_BUFFERS=18
CONFIG_BT_CTLR_TX_BUFFER_SIZE=251
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251

I will try the three parameters you suggested again

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_MAIN_STACK_SIZE=2048
CONFIG_BT_RX_STACK_SIZE.

Besides, I want to ask
When the central receives the notification from the peripheral, the central will go to the notify_func, I print the data I received in the notify_func and use bt_gatt_write to return the information
Is it illegal to do bt_gatt_write in this function? as follow

static uint8_t notify_func(struct bt_conn *conn,
			   struct bt_gatt_subscribe_params *params,
			   const void *data, uint16_t length)
{
	uint32_t mantissa;
	int8_t exponent;
	int err;
	int a=0,b=0;
	int unsub_conn = 0;

	start_time = k_uptime_get_32();

	if (!data) {
		unsub_conn = bt_conn_index(conn);
		printk("[UNSUBSCRIBED] %d %d %d\n",unsub_conn,params->value_handle,subscribe_params[unsub_conn].value_handle);
		subscribe_params[unsub_conn].value_handle=0U;
		 is_connecting = false;
		return BT_GATT_ITER_CONTINUE;
	}
	
	printk("index %d recv %u %u %u %u %u %u %u.\n", bt_conn_index(conn),((uint8_t *)data)[0],((uint8_t *)data)[1],((uint8_t *)data)[2],((uint8_t *)data)[3],((uint8_t *)data)[4],((uint8_t *)data)[5],((uint8_t *)data)[6]);
	
	gatt_write_buf[0] = ((uint8_t *)data)[0];
	gatt_write_buf[1] = count;
	gatt_write_buf[2] =88;
	gatt_write_buf[3] =99;	

	count++;

	write_params[bt_conn_index(conn)].data = gatt_write_buf;
	write_params[bt_conn_index(conn)].length = 8;
	write_params[bt_conn_index(conn)].handle = service_handle;
	write_params[bt_conn_index(conn)].offset = 0;
	write_params[bt_conn_index(conn)].func = write_func;

	err = bt_gatt_write(conn, &write_params[bt_conn_index(conn)]);
	
	if (err) {
		printk("Write failed (err %d)\r\n", err);
	} 
	return BT_GATT_ITER_CONTINUE;
}

Or is it normal to do bt_gatt_write in the while loop of main?

Thanks,

Poyi

[Hung Bui]

I would suggest to do bt_gatt_write() in a work queue instead of calling it a callback. 
You may need to consider increasing the connection interval. You are having 90ms connection interval but having 20 connection and with the ATT_MTU 247 bytes and DLE 251 bytes, if you use all 251 bytes it won't be enough time to serve all 20 connections (4.5ms/connection). 
You may consider turn SLAVE_LATENCY on if not all peripheral need to transmit data at the same time. 

[hmw]

Is there any relationship between the connection interval and the number of peripherals?
What if I want to connect 20 peripherals?

So I set ATT_MTU 247 bytes and DLE 251 bytes, is it actually sending so many bytes?

I thought my write_params.length set to 8 would only send 8 bytes

write_params[bt_conn_index(conn)].length = 8;

You mean that in my prj.conf

CONFIG_BT_RX_BUF_LEN=258
CONFIG_BT_CTLR_TX_BUFFER_SIZE=251
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251
CONFIG_BT_L2CAP_TX_MTU=247
CONFIG_BT_L2CAP_RX_MTU=247

Can these be changed to 8 or 16 bytes, so that my interval can be set smaller?

[Hung Bui]

It will the actual byte you send so if you set the length to 8 the actual number of byte should be 15 bytes (7 bytes overhead). If you don't plan to send large packet, you can reduce the MTU size and the Datalength max. This is given the notification from the peers also don't have large packet size. 

[hmw]

Hello
I modified the following parameters to

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_MAIN_STACK_SIZE=4096
CONFIG_BT_RX_STACK_SIZE=4096

Then an error appeared, as shown in the figure below. What is the cause of this?

<wrn> bt_conn: Disconnected while allocating context

After trying to change these three parameters to 8192, the above error did not appear.

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=8192
CONFIG_MAIN_STACK_SIZE=8192
CONFIG_BT_RX_STACK_SIZE=8192

But there is a Tx Buffer Overflow error, as shown below

[00:36:55.371,032] <err> bt_ctlr_hci: Tx Buffer Overflow
[00:36:55.371,063] <wrn> bt_hci_core: Data buffer overflow (link type 0x01)
[00:36:55.371,063] <err> bt_conn: Unable to send to driver (err -55)

However, it didn’t stop my data transmission, but it looks like it’s not right
For my TX Buffer related settings, I set the official maximum value, as follows

CONFIG_BT_CONN_TX_MAX=18
CONFIG_BT_L2CAP_TX_BUF_COUNT=18
CONFIG_BT_CTLR_RX_BUFFERS=18
CONFIG_BT_CTLR_TX_BUFFERS=18
CONFIG_BT_L2CAP_TX_MTU=247
CONFIG_BT_L2CAP_RX_MTU=247
CONFIG_BT_CTLR_TX_BUFFER_SIZE=251
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251
CONFIG_BT_RX_BUF_LEN=258

How can I modify it to prevent the Data buffer overflow error from happening again?

Thanks.

Poyi

[Hung Bui]

Hi Poyi, 
Please try to increase one stack size at a time. 
I would suggest to increase CONFIG_BT_RX_STACK_SIZE first. The current default value in your project as 1024, correct ? Please try to increase it to 2048 first. 

The next you want to try is CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE. Please try with 4096. I don't think the CONFIG_MAIN_STACK_SIZE need to be increased. 
Please make sure you put all the BLE API calls in to a work queue, or in main thread. 

If you don't plan send large data packet, please set the CONFIG_BT_L2CAP_RX_MTU , CONFIG_BT_CTLR_DATA_LENGTH_MAX to match with your packet size. 

Please let me know until how many peripherals do  you see the "ATT Timeout" error ? What's the data traffic ? How often the peripheral send notification? What's the data size of the notification? 

Please try testing with larger connection interval and with slave latency. 

[hmw]

You mean, I set

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

Come again

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_BT_RX_STACK_SIZE=4096

Is that so?

In addition, currently the smallest can only be set like this

CONFIG_BT_CTLR_DATA_LENGTH_MAX=32
CONFIG_BT_L2CAP_TX_MTU=65
CONFIG_BT_L2CAP_RX_MTU=65

I don’t know if it matches my packet size

Currently my central only uses bt_gatt_write_without_response
And bt_gatt_write_without_response is in cmd_write
The method to change to work_queue is as follows

void write_work_handler(struct k_work *work)
{
int err;
printk("test now_conn %d\n",now_conn);
cmd_write(service_handle, now_conn);
To
}
K_WORK_DEFINE(write_work, write_work_handler);

ATT Timeout appears after a while after connecting with all peripherals, and the time of appearance is random

I don’t really understand the data traffic you mentioned. I thought that after cnetral announced the interval connection, the peripherals would automatically select an interval value to determine how long it would take to send it.

Five of my peripherals will be sent every 100ms, the rest will be sent every 1 second, and 8bytes will be sent every time.

static uint8_t test[8];

rc = bt_gatt_notify(NULL, &hrs_svc.attrs[1], &test, sizeof(test));

test[8] is the data that my peripheral wants to send

and also
Is there any good solution to the Tx Buffer Overflow error mentioned above?

[00:36:55.371,032] <err> bt_ctlr_hci: Tx Buffer Overflow
[00:36:55.371,063] <wrn> bt_hci_core: Data buffer overflow (link type 0x01)
[00:36:55.371,063] <err> bt_conn: Unable to send to driver (err -55)

Or is it related to my CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE and CONFIG_BT_RX_STACK_SIZE?

Thanks,

Poyi

[Hung Bui]

Hi Poyi, 

Please clarify do you still see the error when you change these to:

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_BT_RX_STACK_SIZE=4096

But this suggestion was based on the MPU (memory protection unit) fault, it suggested there was an issue with the memory/stack.

You may think of increasing CONFIG_BT_L2CAP_TX_BUF_COUNT. If you have 20 peripheral and if you try to send a write command to all of the 20 peripherals you may pass the number of 18 tx buffer . 

Your issue seems to be similar to this on going report: https://github.com/zephyrproject-rtos/zephyr/issues/30378 

Could be the same issue. 

Do you have a sniffer ? If you can sniff the whole activity we can check which exact action caused the timeout. 

My concern is that when you have 20 connection, and each connection has an interval of 90ms there are only 4.5ms for each connection and if the scheduler couldn't schedule all the connections good enough you will have packet drop. I would suggest to change the device that sends packet every one second to switch to 1 second interval instead of 90ms. Or at least change them to 500ms interval.