nRF5340 BLE Transmission Stack Buffer Overflow

[AKV]

Hi,

We are developing a mesh network. Our requirement is the one device can connect to three other devices. Using one connection as a peripheral and up to two connections as a central for mesh connections.

We are using nRF5340 DK and the nRF Connect SDK Version 2.6.1.

I am using the function bt_gatt_write_without_response_cb(). The stack buffer size is set with CONFIG_BT_BUF_ACL_TX_COUNT=30.

The CONFIG_BT_BUF_ACL_TX_COUNT is set to 30, allowing 10 buffers for each connection. I'm using a write complete callback to monitor buffer usage. If the buffer usage for any connection exceeds 10 due to transmission issues, I stop writing to that connection until it is cleared.

If one device connects to two other devices, with one being very close and the other being far away and out of range, the stack buffer becomes full for both connections, even for the nearby device. Why does this issue occur, causing difficulty in sending packets to the nearby device when the other device goes out of range?

[Amanda Hsieh]

Hi, 

If one device connects to two other devices, with one being very close and the other being far away and out of range, the stack buffer becomes full for both connections, even for the nearby device. Why does this issue occur, causing difficulty in sending packets to the nearby device when the other device goes out of range?

When a device is out of range, the BLE stack might still try to resend the data, causing the stack's buffers to fill up. This can prevent new data from being queued for transmission, even for other devices that are within range. 

Regards,
Amanda H.

[AKV]

Hi,

Thanks for the response.

Our project currently runs on the nRF52840 with the nRF5 SDK, and we haven't encountered this issue. We are transitioning to the nRF5340 with the NCS to achieve better performance.

However, this buffer overflow issue is significantly affecting our device's performance. Since this is a fundamental problem, could you address it on your end? Additionally, can we expect a solution in the next NCS release?

[Amanda Hsieh]

Are you able to run your app on a 52840 using NCS?

[AKV]

Hi,

Amanda Hsieh said:

Are you able to run your app on a 52840 using NCS?

We're sorry, Our product is scheduled to be delivered to the customer at the end of this month, and we're currently focused on development and testing. Unfortunately, no time to handle the porting and testing on the nRF52840.

We are confident that this is an NCS-related issue, as we did not encounter this problem when using the nRF5 SDK.

AKV said:

Please find the attached log with enabled log.

CONFIG_BT_GATT_LOG_LEVEL_DBG=y
CONFIG_BT_ATT_LOG_LEVEL_DBG=y
CONFIG_BT_L2CAP_LOG_LEVEL_DBG=y

nRF5340_MultiConnection_BufferOverflow_issue.txt

As you requested I have attached the logs. Were you able to verify the log? 

[Amanda Hsieh]

Hi,

Could you try to increase the following kconfigs for setting per-link ACL buffers in SDC?

CONFIG_BT_CTLR_SDC_TX_PACKET_COUNT
CONFIG_BT_CTLR_SDC_RX_PACKET_COUNT

As documented there, increasing these values is not generally useful unless CPU load is very high.

-Amanda H.

[AKV]

Hi,

I attempted to increase the values for CONFIG_BT_CTLR_SDC_TX_PACKET_COUNT to 4 and CONFIG_BT_CTLR_SDC_RX_PACKET_COUNT to 4. But the issue persists. I've attached the log for your reference.

writeCmdTxCmpltCallback index:1 ConnHandle:0, OutIndex:90
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:164
writeCmdTxCmpltCallback index:1 ConnHandle:0, OutIndex:91
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:165
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:166
writeCmdTxCmpltCallback index:1 ConnHandle:0, OutIndex:92
writeCmdTxCmpltCallback index:1 ConnHandle:0, OutIndex:93
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:167
writeCmdTxCmpltCallback index:1 ConnHandle:0, OutIndex:94
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:168
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:169
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:170
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:171
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:172
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
TX[1] BufferFull:1, ConnHandle:0, InIndex:104, OutIndex:94, StackBufferUserd:10
[CONN]: ####### DISCONNECTED Reason: 8 #######
[CONN]: CONN_COUNT:1, conn_handle:0, conn->state:0, k_uptime_get_32():11523
TX[0] BufferFull:1, ConnHandle:3, InIndex:182, OutIndex:172, StackBufferUserd:10
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:173
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:174
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:175
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:176
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:177
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:178
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:179
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:180
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:181
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:182
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:183
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:184
writeCmdTxCmpltCallback index:0 ConnHandle:3, OutIndex:185

As you can see, the device with connection handle 0 gets disconnected, and after turning it off, the buffer for the device with connection handle 1 also starts to fill up.

The log shows that after turning off connection handle 0, the first 4 buffer full logs are associated with connection handle 0, then the buffer full issue occurs for the near by device connection handle 3 as well.

The supervision timeout is set to 3 seconds, so if one device goes out of range, it affects that all nearby devices are also unable to send packets until that device is disconnected.

[AKV]

Hi, Amanda Hsieh 

Are there any updates?

We are running short on time. Please help to resolve the issue as quickly as possible.

[Amanda Hsieh]

Hi, 

We suspect the issue is related to #332192 Vida working on. It turned out to be a critical bug in the Zephyr host. I would suggest fixing the #332192 as Vida's suggestions, then that might also fix the issues in this case. 

-Amanda H.