NCS Nordic UART service: BT RX HCI timeout

RE: NCS Nordic UART service: BT RX HCI timeout

Tom_H — Tue, 02 Aug 2022 14:33:49 GMT

I have created a simplified version to reproduce the problem. Here is the link to the private case:
devzone.nordicsemi.com/.../293262

RE: NCS Nordic UART service: BT RX HCI timeout

Hung Bui — Fri, 22 Jul 2022 08:09:07 GMT

Please create a new Private case to share the source code. Please send me here the link to the case after you create it.

RE: NCS Nordic UART service: BT RX HCI timeout

Tom_H — Fri, 22 Jul 2022 06:51:49 GMT

Thank you very much!

I will try to create a simplified version of my code. The code is for a customer project and cannot be published. How should I provide the sample code?

RE: NCS Nordic UART service: BT RX HCI timeout

Hung Bui — Thu, 21 Jul 2022 13:19:00 GMT

Hi Thomas,

We will need to wait for patch for those issues I linked to. We will work on it shortly.
If you can provide a simplified version of your code that we can reproduce the issue here it would be great (both central and peripheral code)

RE: NCS Nordic UART service: BT RX HCI timeout

Tom_H — Thu, 21 Jul 2022 10:39:33 GMT

At maximum request rate (without delay between requests) the problem is reproducible even with a connection interval of 75 ms.

I have tried to increase the CONFIG_BT_L2CAP_TX_BUF_COUNT to 128, which seems to improve the situation somewhat, but does not solve the problem. I think the problem is more related to the RX buffering since the uplink traffic and payload size from the peripheral to the central is much higher.

A note on the threading context, the bt_nus_client_send() function is called from a separate thread which has thread priority 7.

Do you have any further suggestions?

RE: NCS Nordic UART service: BT RX HCI timeout

Hung Bui — Mon, 18 Jul 2022 14:40:58 GMT

Here is another related issue: https://github.com/zephyrproject-rtos/zephyr/issues/34600

Could you try the suggested workaround by increasing CONFIG_BT_L2CAP_TX_BUF_COUNT and

CONFIG_BT_L2CAP_TX_BUF_COUNT ? Try, say 128.

Of course the draw back is the RAM consumption.

RE: NCS Nordic UART service: BT RX HCI timeout

Hung Bui — Mon, 18 Jul 2022 14:14:44 GMT

Hi Thomas,

We are suspecting you are having the same issue as reported here: https://github.com/zephyrproject-rtos/zephyr/issues/25917#issuecomment-639348125
I'm checking if there is any workaround.
Could you confirm that at 75ms interval , the problem is not reproducible ?

RE: NCS Nordic UART service: BT RX HCI timeout

Tom_H — Mon, 18 Jul 2022 09:10:41 GMT

Thank you!

The CONFIG_ASSERT option is already set in an overlay file for my debug build and the error above is from a debug build.

# Optimize for debug
CONFIG_DEBUG_OPTIMIZATIONS=y
CONFIG_DEBUG_THREAD_INFO=y
CONFIG_ASSERT=y

# Enable Thread awareness for Debugging in Segger Ozone
CONFIG_THREAD_ANALYZER=y
CONFIG_THREAD_NAME=y

I have tested bt_gatt_write_without_response() and it shows the same behaviour as bt_gatt_write_without_response_cb(). It only works with reduced request rate.

Update:
I have also tested disabling the BT scanner when maximum number of 20 connections are established. And added a log message when the tx_ctx get released in the on_sent_wo_rsp() method. Unfortunately, the app crashes in the same way. Here ist the log output.

RE: NCS Nordic UART service: BT RX HCI timeout

Hung Bui — Fri, 15 Jul 2022 12:25:01 GMT

Hi Thomas,
I'm checking internally to see if R&D has any advice on this.

Have you checked if you use bt_gatt_write_without_response() do you have any issue ?
I suspecting there is a messing up when the central waiting for the response from each of the peripheral.

Please try with CONFIG_ASSERT=y so we have more log.

RE: NCS Nordic UART service: BT RX HCI timeout

Tom_H — Thu, 14 Jul 2022 12:42:32 GMT

Thanks for the quick reply, I have tested your suggestion.

I think it is good to describe the problem in more detail. The connected peripherals have a continuous data stream to the centrail, and the error occurs when a user-initiated request is sent from the central to the connected peripherals and they respond. The error occur imedently after the request is sent, I think when the response is received.

To reduce the rate of sending data from each of the peripherals down, I have reduced the request rate by inserting a 200 ms delay between the requests.

The Firmware runs on two different systems with different configurations. Here are the configuration and test results:

System 1:

connection interval 75 ms (min & max)
packet generation with fixed rate of ~3.5 Hz
- approx. 220 packets per minute
- approx. 6 kbit/s per connection
scan parameters:
- Intervall: 500 ms
- Window: 40 ms
Test 1 - Reduced request rate: works with a request delay of 200 ms for 20 connections
Test 2 - CONFIG_BT_CTLR_RX_BUFFERS=10: does not work

System 2:

connection interval 25 ms (min & max)
none deterministic packet generation
- approx. 40 packets per minute
- approx. 0.4 kbit/s per connection
scan parameters:
- Intervall: 250 ms
- Window: 31.25 ms
Test 1 - Reduced request rate: does not work (request delay of 200 ms)
Test 2 - CONFIG_BT_CTLR_RX_BUFFERS=10: does not work
Error occur more often than with system 1

Or could it somehow be related with the context management in my multi-NUS service implementation. This function is called when a request is sent to the connected devices.

int multi_nus_t::send_broadcast(uint8_t* buf, size_t len) {
    int err = 0;

    /* How many connections are there in the Connection Context Library */
    const size_t num_nus_conns = bt_conn_ctx_count(&conns_ctx_lib);

    LOG_DBG("Send Broadcast");
    for (size_t i = 0; i < num_nus_conns; i++) {
        const struct bt_conn_ctx* ctx = bt_conn_ctx_get_by_id(&conns_ctx_lib, i);

        if (ctx) {
            struct bt_nus_client* nus_client = (bt_nus_client*) ctx->data;

            if (nus_client) {
                // copy payload so it persists throught the BLE stack
                uint8_t* ble_data = (uint8_t*) k_malloc(len);

                if (ble_data == NULL) {
                    LOG_ERR("Could not allocate broadcast data connection %d", i);
                    goto release_ctx;
                }
                memcpy(ble_data, buf, len);

                err = bt_nus_client_send(nus_client, ble_data, len);
                if (err) {
                    LOG_ERR("Failed to send data over BLE connection %d (err %d)", i, err);
                    // return the semaphore as sending was not successful
                    k_free(ble_data);
                } else {
                    LOG_INF("Sent broadcast to connection %d: %d bytes", i, len);
                    k_msleep(200); // delay to reduce the request rate
                }
            }
            release_ctx:
            bt_conn_ctx_release(&conns_ctx_lib, (void*) ctx->data);
        }
    }
    return err;
}

Do you have any further suggestions?

RE: NCS Nordic UART service: BT RX HCI timeout

Hung Bui — Wed, 13 Jul 2022 12:18:19 GMT

Hi Thomas,

To isolate the issue if it's a memory overflow issue or not, could you try to reduce the rate of sending data from each of the peripheral down and check if you can handle above 10 connections ?

Could you also try to increase CONFIG_BT_CTLR_RX_BUFFERS to 10 for example ?

Please provide the information on:

- Connection interval

- How long it would take for the error to occur ?