In multiple central connection, BLE Tx complete or on_sent callback not received for other peripheral devices when one of the peripheral device is out of range or reset.

[MEHUL RANA]

SDK: nRF Connect SDK v2.6.1
Hardware: nRF52840 (Central Device) and nRF52832 (Peripheral Devices)

Detailed Scenario:
The central device (nRF52840) has 8 peripheral devices connected to it.
Central is sending data to each peripheral device at connection interval of 188ms. The supervision timeout is set to 24 seconds.
When one of the peripheral devices undergoes a power reset, or goes out of range, BLE link breaks.
But central has not receive disconnection event yet. So, central keep sending data to this device using bt_gatt_write_without_response_cb function. During this central fails to get BLE tx complete event or on_sent callback for other 7 peripheral which are connected.

To send data from the central to peripheral devices, I am using the following function:

int bt_nus_client_send(struct bt_nus_client nus_c, const uint8_t data, uint16_t len)
{
	int err;

	if (!nus_c->conn) 
	{
		return -ENOTCONN;
	}

	err = bt_gatt_write_without_response_cb(nus_c->conn, nus_c->handles.rx, data, len, false, on_sent, nus_c);
	if (err) 
	{
		atomic_clear_bit(&nus_c->state, NUS_C_RX_WRITE_PENDING);
	}

	return err;
}

Expected Behavior:

The central device should be able to send Data to the remaining connected peripheral devices during one of the peripheral device is out of range or reset.

Note:

bt_nus_client_send() not return any err code during this Scenario.

I have captured sniffer data and confirmed that other devices are not able to receive data, but they send/receive empty PDUs during this issue.

Thank you for your assistance.
Best regards,
Mehul

[Kenneth]

Hello,

Please see a similiar discussion here:
Multi-NUS: Sending several messages in a row 

Kenneth

[MEHUL RANA]

Hi Kenneth ,

The above link you mentioned is not relevant for my issue.

I am able to send data to all connected peripheral devices with out loss.

The issue occurs only when one of the peripheral devices in the network undergoes a power reset. During this reset, the central device does not receive the disconnect event until the supervision timeout elapses, which is 24 seconds. After received disconnect event , the system functions perfectly and communication with all connected peripherals resumes without any data loss.

Can you please explain this behavior why i am not able to send data to other connected peripheral when one device in network is out of range(supervision timeout) and not getting callback from bt_gatt_write_without_response_cb() function.

[Kenneth]

Hello again, 

I believe the linked case is close to the same problem, it does say for instance in the other case: "You can't send a broadcast to all peripherals in "parallel" because the system waits for each message to be finished (semaphore)."

You experience the same problem, since one of the connections are waiting for a supervisor timeout. The linked case discuss one way to solve this using unique semaphores for each link. I don't see a different way of solving this.

Kenenth

[Kenneth]

If you still believe the problem is not related, then I think you need to share with me how you are calling bt_nus_client_send(). For instance are there multiple calls to bt_nus_client_send() for the same link in a row? Is each call to bt_nus_client_send() only called after callbacks? Or is each link independantly calling bt_nus_client_send()? How do you handle bt_nus_client_send() if it return an error code (try again or go to next link)?

Kenneht

[MEHUL RANA]

Hi Kenneth ,

Sure, Here i attached relevant code,

/***************************************************************************/
//NUS  file
/***************************************************************************/

static void on_sent(struct bt_conn *conn, void *user_data)
{
	struct bt_nus_client *nus_c;
	const void *data;
	uint16_t length;

	nus_c = user_data;

	/* Make a copy of volatile data that is required by the callback. */
	data = nus_c->rx_write_params.data;
	length = nus_c->rx_write_params.length;

	atomic_clear_bit(&nus_c->state, NUS_C_RX_WRITE_PENDING);
	if (nus_c->cb.sent) {
		nus_c->cb.sent(nus_c, 0, data, length);
	}
}


int bt_nus_client_send(struct bt_nus_client *nus_c, const uint8_t *data, uint16_t len)
{
	int err;

	if (!nus_c->conn) {
		return -ENOTCONN;
	}

	err = bt_gatt_write_without_response_cb(nus_c->conn, nus_c->handles.rx, data, len, false, on_sent, nus_c);
	if (err) {
		atomic_clear_bit(&nus_c->state, NUS_C_RX_WRITE_PENDING);
	}

	return err;
}

/***************************************************************************/
// Central service
/***************************************************************************/

#define CSH_MAX_CONN_BUFF				10			// Maximum tx buffer allow for each child connection
#define CSH_MAX_PACKET_ALLOW_IN_BUFF	54			// Total tx buffer allocated among all children

//CONFIG_BT_MAX_CONN = 8

static struct bt_nus_client _bcNusClient[CONFIG_BT_MAX_CONN];

static uint8_t _pktInSendBuff[CONFIG_BT_MAX_CONN];
static uint8_t _totalPktInSendBuff;
/***************************************************************************/
static void CSHi_CentralDataSent(struct bt_nus_client *nus, uint8_t err,
					const uint8_t *const data, uint16_t len)
{
	uint16_t connIndx = bt_conn_index(nus->conn);

	NRF_LOG_INFO("CSHi_CentralDataSent cb for :%d",connIndx);

	if (_pktInSendBuff[connIndx])
	{
		_pktInSendBuff[connIndx]--;
		if (_pktInSendBuff[connIndx] == 0)
		{
			GCD_SendPendingDataForConn(connIndx);
		}
	}

	if (_totalPktInSendBuff)
	{
		_totalPktInSendBuff--;
	}
	
	if (err) 
	{
		NRF_LOG_WARNING("ATT error code: 0x%02X", err);
	}
}

/***************************************************************************/
ret_code_t CSH_WriteOnBccService(const uint8_t* pData, uint16_t dataLength, uint16_t connIndx)
{
	int32_t errCode = NRF_ERROR_NOT_FOUND;

	if ((connIndx != BLE_CONN_HANDLE_INVALID) && (_bcNusClient[connIndx].conn))
	{
		if (((_pktInSendBuff[connIndx] < CSH_MAX_CONN_BUFF) && (_totalPktInSendBuff < CSH_MAX_PACKET_ALLOW_IN_BUFF)))
		{
			errCode = bt_nus_client_send(&_bcNusClient[connIndx], pData, dataLength);
			if (errCode) 
			{
				NRF_LOG_ERROR("Failed to send data over BLE connection %d""(err %d)", connIndx, errCode);
			} 
			else 
			{
				_pktInSendBuff[connIndx]++;
				_totalPktInSendBuff++;
			}	
		}
		else
		{
			NRF_LOG_ERROR("NUS send buffer full for connection: %d", connIndx);
			errCode = NRF_ERROR_BUSY;
		}
	}
	else
	{
		NRF_LOG_ERROR("Data send failed. Invalid BLE connection: %d", connIndx);
	}		

	return errCode;
}


/***************************************************************************/
static ret_code_t CSHi_InitBccService(struct bt_conn *conn, uint16_t connIndx)
{
	int32_t errCode;

	struct bt_nus_client_init_param init = {
		.cb = {
			.received = CSHi_CentralDataReceived,
			.sent = CSHi_CentralDataSent,
		}
	};

	memset(&_bcNusClient[connIndx], 0, sizeof(_bcNusClient[connIndx]));

	errCode = bt_nus_client_init(&_bcNusClient[connIndx], &init);
	if (errCode) 
	{
		NRF_LOG_ERROR("NUS Client initialization failed (err %d)", errCode);
		return errCode;
	}

	return errCode;
}


/***************************************************************************/
//Data handler
/***************************************************************************/

static void GND_HandleBroadcastPacket(const uint8_t* pData, uint16_t dataLength)
{
	uint16_t slot;

	for (slot = 0; slot < CONFIG_BT_MAX_CONN; slot++)
	{
		if (_connDevice[slot].connHandle != BLE_CONN_HANDLE_INVALID)
		{
			CSH_WriteOnBccService(pData, dataLength, _connDevice[slot].connHandle)
		}

	}
}

/***************************************************************************/

Code Explanation:

bt_nus_client_send() Calls:

The bt_nus_client_send function is called through CSH_WriteOnBccService.
Before calling bt_nus_client_send, it checks if the connection is valid and if the send buffer has space.

Handling Multiple Calls:
Each call to bt_nus_client_send is managed independently for each link.
Buffer counts (_pktInSendBuff and _totalPktInSendBuff) ensure that multiple calls for the same link are controlled.

Callback Handling:
The on_sent callback triggers the CSHi_CentralDataSent callback.
The CSHi_CentralDataSent callback manages packet counts and retries sending pending data if buffers are cleared.

/Mehul

[Kenneth]

Hello,

What I suspect is happening here is that you are calling bt_gatt_write_without_response_cb() several times in a row for the same connection, without the data is actually sent for that connection (because the peer is not available, e.g. turned OFF). Eventually the following will happen: "This function will block while the ATT request queue is full".

This was found in the gatt.h where you can find the api documentation for bt_gatt_write_without_response_cb(). So in this case the bt_gatt_write_without_response_cb() will not return until buffer is available or link loss.

Kenneth

[MEHUL RANA]

Hi Kenneth ,
In my implementation, each connection is limited to a maximum of 10 pending callbacks (CSH_MAX_CONN_BUFF = 10). This ensures that the bt_nus_client_send() function will not be called once there are 10 pending callbacks for that connection.

I have set the following configurations in the prj.conf file:

CONFIG_BT_CONN_TX_MAX=65
CONFIG_BT_L2CAP_TX_BUF_COUNT=65

These settings provide sufficient buffer space, allowing for more callbacks even if one of the disconnected peers uses 10 buffers. Therefore, there should still be available buffers for other connections, preventing the system from blocking.

If i understand it wrong, please correct me.

One more thing, The bt_gatt_write_without_response_cb() function should return an -ENOMEM (-12) error code when the queue is full, but it does not return any error code.

Best regards,
Mehul

[Kenneth]

Can you make the CSH_MAX_CONN_BUFF much less than 10 for test, e.g. 3 for test. This could at least narrow down if we are on the right track. If 10 fails and 3 works, then I can look into the details more.

Kenneth

[MEHUL RANA]

Yes I have tested CSH_MAX_CONN_BUFF with different values CSH_MAX_CONN_BUFF=1 and 2 worked as expected. But 3 is not working.

/Mehul