BLE multilink ATT timeout problem occurs when the number of connections increases

[hmw]

Recently I am using BLE in Zephyr to implement the multilink central function
However, none of the references currently have a similar function, and there is also the nordic zephyr code
https://github.com/nrfconnect/sdk-nrf  <-There are only examples of multilink peripherals in this link
So I wrote one to implement it. The goal I want to achieve is one central to 30 peripherals
The current connection with 4 peripherals is fully stable, but when I increase the connected peripherals, the following error will appear

<err> bt_attLATT Timeout
<wrn> bt_att: No ATT channel for MTU 5
<wrn> bt_att: No pending ATT request

The picture below shows the error when connecting 20 peripherals

At present, I know that modifying the Interval connection will indeed improve, but only to make <err> bt_attLATT Timeout happen later.

How can I avoid this problem so that I can connect 30 peripherals stably?

Below is my code
Or you can go to this page to download https://github.com/mfinmuch/zephyr-ble-mulrilink-test

3225.multilink central.rar

Thanks,

Poyi

[Hung Bui]

Hi Hmw, 

Could you describe a little bit more about your application ? 
What's the connection interval you used ? What data do you send/receive , which direction ? 

You actually got a MPU fault, suggesting there could be a stack overflow. Please try increasing this: CONFIG_BT_RX_STACK_SIZE. Try double the stack size, maybe to 4096. 
Also, try to increase these if it's already as follow: 

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_MAIN_STACK_SIZE=2048

Are you using Zephyr stack or Nordic stack ?

[hmw]

Currently I set the interval as follows

#define MIN_CONNECTION_INTERVAL         72     
#define MAX_CONNECTION_INTERVAL         72    
#define SLAVE_LATENCY                   0    
#define SUPERVISION_TIMEOUT             50 

I use the BLE example central_hr and peripheral provided by Zephyr to make changes, it should be Zephyr stack

My central will look for peripherals first
Establish a connection, peripherals will send data to central(use notify), and central will return data to it (use bt_gatt_write)

And the data is just a simple sequence

I thought only the following parameters need to be set

CONFIG_HEAP_MEM_POOL_SIZE=4096
CONFIG_BT_RX_BUF_LEN=258
CONFIG_BT_ATT_TX_MAX=10
CONFIG_BT_ATT_PREPARE_COUNT=10
CONFIG_BT_CONN_TX_MAX=18
CONFIG_BT_L2CAP_TX_BUF_COUNT=18
CONFIG_BT_L2CAP_TX_MTU=247
CONFIG_BT_L2CAP_RX_MTU=247
CONFIG_BT_L2CAP_DYNAMIC_CHANNEL=y
CONFIG_BT_CTLR_PHY_2M=y
CONFIG_BT_CTLR_RX_BUFFERS=18
CONFIG_BT_CTLR_TX_BUFFERS=18
CONFIG_BT_CTLR_TX_BUFFER_SIZE=251
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251

I will try the three parameters you suggested again

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_MAIN_STACK_SIZE=2048
CONFIG_BT_RX_STACK_SIZE.

Besides, I want to ask
When the central receives the notification from the peripheral, the central will go to the notify_func, I print the data I received in the notify_func and use bt_gatt_write to return the information
Is it illegal to do bt_gatt_write in this function? as follow

static uint8_t notify_func(struct bt_conn *conn,
			   struct bt_gatt_subscribe_params *params,
			   const void *data, uint16_t length)
{
	uint32_t mantissa;
	int8_t exponent;
	int err;
	int a=0,b=0;
	int unsub_conn = 0;

	start_time = k_uptime_get_32();

	if (!data) {
		unsub_conn = bt_conn_index(conn);
		printk("[UNSUBSCRIBED] %d %d %d\n",unsub_conn,params->value_handle,subscribe_params[unsub_conn].value_handle);
		subscribe_params[unsub_conn].value_handle=0U;
		 is_connecting = false;
		return BT_GATT_ITER_CONTINUE;
	}
	
	printk("index %d recv %u %u %u %u %u %u %u.\n", bt_conn_index(conn),((uint8_t *)data)[0],((uint8_t *)data)[1],((uint8_t *)data)[2],((uint8_t *)data)[3],((uint8_t *)data)[4],((uint8_t *)data)[5],((uint8_t *)data)[6]);
	
	gatt_write_buf[0] = ((uint8_t *)data)[0];
	gatt_write_buf[1] = count;
	gatt_write_buf[2] =88;
	gatt_write_buf[3] =99;	

	count++;

	write_params[bt_conn_index(conn)].data = gatt_write_buf;
	write_params[bt_conn_index(conn)].length = 8;
	write_params[bt_conn_index(conn)].handle = service_handle;
	write_params[bt_conn_index(conn)].offset = 0;
	write_params[bt_conn_index(conn)].func = write_func;

	err = bt_gatt_write(conn, &write_params[bt_conn_index(conn)]);
	
	if (err) {
		printk("Write failed (err %d)\r\n", err);
	} 
	return BT_GATT_ITER_CONTINUE;
}

Or is it normal to do bt_gatt_write in the while loop of main?

Thanks,

Poyi

[Hung Bui]

I would suggest to do bt_gatt_write() in a work queue instead of calling it a callback. 
You may need to consider increasing the connection interval. You are having 90ms connection interval but having 20 connection and with the ATT_MTU 247 bytes and DLE 251 bytes, if you use all 251 bytes it won't be enough time to serve all 20 connections (4.5ms/connection). 
You may consider turn SLAVE_LATENCY on if not all peripheral need to transmit data at the same time. 

[hmw]

Is there any relationship between the connection interval and the number of peripherals?
What if I want to connect 20 peripherals?

So I set ATT_MTU 247 bytes and DLE 251 bytes, is it actually sending so many bytes?

I thought my write_params.length set to 8 would only send 8 bytes

write_params[bt_conn_index(conn)].length = 8;

You mean that in my prj.conf

CONFIG_BT_RX_BUF_LEN=258
CONFIG_BT_CTLR_TX_BUFFER_SIZE=251
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251
CONFIG_BT_L2CAP_TX_MTU=247
CONFIG_BT_L2CAP_RX_MTU=247

Can these be changed to 8 or 16 bytes, so that my interval can be set smaller?

[Hung Bui]

It will the actual byte you send so if you set the length to 8 the actual number of byte should be 15 bytes (7 bytes overhead). If you don't plan to send large packet, you can reduce the MTU size and the Datalength max. This is given the notification from the peers also don't have large packet size. 

[hmw]

Hello
I modified the following parameters to

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_MAIN_STACK_SIZE=4096
CONFIG_BT_RX_STACK_SIZE=4096

Then an error appeared, as shown in the figure below. What is the cause of this?

<wrn> bt_conn: Disconnected while allocating context

After trying to change these three parameters to 8192, the above error did not appear.

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=8192
CONFIG_MAIN_STACK_SIZE=8192
CONFIG_BT_RX_STACK_SIZE=8192

But there is a Tx Buffer Overflow error, as shown below

[00:36:55.371,032] <err> bt_ctlr_hci: Tx Buffer Overflow
[00:36:55.371,063] <wrn> bt_hci_core: Data buffer overflow (link type 0x01)
[00:36:55.371,063] <err> bt_conn: Unable to send to driver (err -55)

However, it didn’t stop my data transmission, but it looks like it’s not right
For my TX Buffer related settings, I set the official maximum value, as follows

CONFIG_BT_CONN_TX_MAX=18
CONFIG_BT_L2CAP_TX_BUF_COUNT=18
CONFIG_BT_CTLR_RX_BUFFERS=18
CONFIG_BT_CTLR_TX_BUFFERS=18
CONFIG_BT_L2CAP_TX_MTU=247
CONFIG_BT_L2CAP_RX_MTU=247
CONFIG_BT_CTLR_TX_BUFFER_SIZE=251
CONFIG_BT_CTLR_DATA_LENGTH_MAX=251
CONFIG_BT_RX_BUF_LEN=258

How can I modify it to prevent the Data buffer overflow error from happening again?

Thanks.

Poyi

[Hung Bui]

Hi Poyi, 
Please try to increase one stack size at a time. 
I would suggest to increase CONFIG_BT_RX_STACK_SIZE first. The current default value in your project as 1024, correct ? Please try to increase it to 2048 first. 

The next you want to try is CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE. Please try with 4096. I don't think the CONFIG_MAIN_STACK_SIZE need to be increased. 
Please make sure you put all the BLE API calls in to a work queue, or in main thread. 

If you don't plan send large data packet, please set the CONFIG_BT_L2CAP_RX_MTU , CONFIG_BT_CTLR_DATA_LENGTH_MAX to match with your packet size. 

Please let me know until how many peripherals do  you see the "ATT Timeout" error ? What's the data traffic ? How often the peripheral send notification? What's the data size of the notification? 

Please try testing with larger connection interval and with slave latency.