BLE throughput differs between NCS v2.6.0 and v2.9.2.

[chiba]

Hi,

BLE throughput differs between NCS v2.6.0 and v2.9.2.
v2.9.2 appears to have fewer MoreData packets.
(Both have CONFIG_BT_CTLR_SDC_MAX_CONN_EVENT_LEN_DEFAULT set to 7500)

In v2.9.2, I confirmed that setting `CONFIG_BT_CTLR_SDC_MAX_CONN_EVENT_LEN_DEFAULT=15000` changes the number of more data packets.
Is there a difference in how `CONFIG_BT_CTLR_SDC_MAX_CONN_EVENT_LEN_DEFAULT` is handled between v2.6.0 and v2.9.2?

- Using the nrf5340
- Configured for multi-protocol communication with Matter (At the time of this test, commissioning had not been performed, and no Matter communication was taking place)

[Samruddhi Jadhav]

Hello,

Can you please upload your whole application in v2.9.2?

Best Regards,

Samruddhi

[chiba]

HI, I've created a project so you can reproduce it on a DK board.

Overview

Packets are periodically transmitted from the Central to the Peripheral.
Once both devices are powered on, they connect automatically and transmit packets at 1-second intervals.

Project Details

• central_52840dk
  - Description: Central-side project for the nRF52840 DK.
  - Base: ncs292/zephyr/samples/bluetooth/central
• lock_5340dk_ncs260
  - Description: Peripheral-side project for the nRF5340 DK (NCS 2.6.0).
  - Base: ncs260/nrf/samples/matter/lock
• lock_5340dk_ncs292
  - Description: Peripheral-side project for the nRF5340 DK (NCS 2.9.2).
  - Base: ncs292/nrf/samples/matter/lock

8055.lock_5340dk_ncs292.zip

7624.lock_5340dk_ncs260.zip

1856.central_52840dk.zip

[Edvin]

Hello,

Is the ncs260 device and ncs292 device running on the same exact HW, and under similar conditions? The reason I ask is that I can't seem to see any difference in behavior, but there seems to be a bit more packet loss in the latter trace.

That said, it seems like both traces show a chunk of 158 byte long messages. Sometimes 6 packets, sometimes 5 and sometimes 4. Not an awful lot of data. 

Do you mean that they should always be the same? E.g. 5 or 6 packets? 

Where in your application (both in 2.6.0 and 2.9.2) do you transmit this data? 

Regardless of packet loss, the data should go through, as long as it is queued with a return value of Success (0). 

Can you confirm that this is the case? That you check the return value when you queue up the data for TX?

Best regards,

Edvin

[chiba]

Hi,

Both ncs2.6.0 and ncs2.9.2 are being operated on the same DK board. I have sent the project files, so could you please perform a packet capture on the Nordic side as well?

Here is the background regarding this issue:

1. We discovered that the throughput differs between ncs2.6.0 and ncs2.9.2 in a certain product, with ncs2.9.2 being slower. After capturing the BLE packets, we confirmed that the number of "More Data" packets differs between ncs2.6.0 and ncs2.9.2. This was the starting point of the DevZone topic.

2. Samruddhi Jadhav requested that we provide the project.

3. I have created and provided a test project for the DK board so that the issue can be reproduced on your end.

[Edvin]

Hello,

Yes. Samruddhi asked if I could have a look at your ticket. 

Can you please clarify for me where you are sending the packets? The reason I ask is that most of us who are familiar with Bluetooth are not that familiar with the Matter stack, and since this is all written in C++, I need some help navigating. So can you please point me to where you are sending/queuing the packets, so that I can speed up the investigation to help you solve your issues quicker?

Best regards,

Edvin

[chiba]

Hi,

This project is based on a Matter sample project, but it concerns Bluetooth communication.
Matter is not running, and commissioning has not been performed.
Additionally, the project I provided is a modified version of the sample project provided by NCS, but the changes are minimal.
Please compare it with the original NCS sample code.

Project Details (Repeat)

central_52840dk
- Description: Central-side project for the nRF52840 DK.
- Base: ncs292/zephyr/samples/bluetooth/central
lock_5340dk_ncs260
- Description: Peripheral-side project for the nRF5340 DK (NCS 2.6.0).
- Base: ncs260/nrf/samples/matter/lock
lock_5340dk_ncs292
- Description: Peripheral-side project for the nRF5340 DK (NCS 2.9.2).
- Base: ncs292/nrf/samples/matter/lock

[central_52840dk]---ble_packet--->[lock_5340dk_ncs260]

[central_52840dk]---ble_packet--->[lock_5340dk_ncs290]

[Edvin]

I am struggling to find where you are sending the large arrays containing 0x00's. Perhaps you can help me locate them.

BR,

Edvin

[chiba]

Hi,

> sending the large arrays containing 0x00's'

I believe this is the code on the central_52840dk side.

static void timeout_handler( struct k_timer *timer )
{
    static uint8_t data[155] ={0};
    static uint16_t count=0;
	
    if (fts_rx_handle) {
        printk("send sequence %d\n", count);
        data[0] = count++;
        bt_fts_client_send(data, sizeof(data));
        bt_fts_client_send(data, sizeof(data));
        bt_fts_client_send(data, sizeof(data));
        bt_fts_client_send(data, sizeof(data));
        bt_fts_client_send(data, sizeof(data));
        bt_fts_client_send(data, sizeof(data));
        bt_fts_client_send(data, sizeof(data));
        bt_fts_client_send(data, sizeof(data));
    } else {
        printk("Waiting for handle discovery...\n");
    }
}

```

This packet is received by either lock_5340dk_ncs260 or lock_5340dk_ncs292.

[central_52840dk]---ble_packet--->[lock_5340dk_ncs260]

[central_52840dk]---ble_packet--->[lock_5340dk_ncs292]

The sender is central_52840dk.
The receiver is either lock_5340dk_ncs260 or lock_5340dk_ncs292.
lock_5340dk_ncs292 has fewer MD packets.

It appears that lock_5340dk_ncs292 is not returning an ACK.

[Edvin]

Hello,

So I looked at your applications, and captured a sniffer trace between the lock_5340_ncs292 and the central_52840dk applications, and this is what I got in my office environment:

new_trace_292.pcapng

From what I can tell, there is no degrade in performance. From your ...292.pcapng trace I saw that there wass a lot of packet loss, but the traces themselves can't tell why the central didn't try to send more packets. 

If you do have a certain amount of data that you need to send, you need to check the return value of your bt_fts_client_send_data(), which does indeed return a value:

/* Function to send data to the discovered FTS RX handle */
int bt_fts_client_send(const uint8_t *data, uint16_t len)
{
	if (!btconn || !fts_rx_handle) {
		printk("FTS RX handle not discovered or not connected\n");
		return -EIO;
	}

	int err = bt_gatt_write_without_response(btconn, fts_rx_handle, data, len, false);
	printk("bt_gatt_write_without_response len:%d 0x%04x: %d\n", len, fts_rx_handle, err);

	return err;
}

So if it returns 0, it means that the packet was successfully queued, and will be transmitted (when the link has capacity). 

If it returns anything other than 0, it means that the packet was not queued, and will not be sent. In those cases, you need to look at the return value to see why these were not able to be queued.

I see from your 262 trace that it always sent 5 packets, so I suspect that you did change the central application after capturing this trace, before sending the central application, since the central application always sends 8 packets once every second.

Best regards,

Edvin

[chiba]

Hi,

I believe the underlying throughput issue stems from differences in MD packets.
It appears that the NCS292 can transmit fewer MD packets per CI.
The sample project I provided is intended solely for verifying these differences in MD packets.
I am not concerned with throughput or data loss in this project.

> From your ...292.pcapng trace I saw that there wass a lot of packet loss, but the traces themselves can't tell why the central didn't try to send more packets.

This is exactly what I want to check.

[Edvin]

I realized something.

While yes, there were retransmissions, I thought it was because the radio was reserved for Thread, and perhaps the default value did change between 2.6.2 and 2.9.2.

Try adding this to your lock_5340_292 application:

sysbuild\ipc_radio\boards\nrf5340dk_nrf5340_cpunet.conf:

CONFIG_BT_CTLR_SDC_MAX_CONN_EVENT_LEN_DEFAULT_OVERRIDE=y
CONFIG_BT_CTLR_SDC_MAX_CONN_EVENT_LEN_DEFAULT=20000

What this does is that it tells the device that it can spend more time each connection interval handling one BLE connection. In this case 20ms. It does not mean it will spend 20ms in cases where there is no more data being sent. It does however mean that when there is a lot of data on BLE, it will spend less data on Thread, leading to poorer Matter performance. Retransmissions will handle this, so you shouln't ultimately loose any packets. It is simply a tradeoff whether you want to prioritize radio time on BLE or Matter.

Using this, I see that all 8 packets are sent within one connection interval, as you can see in the attached trace, but feel free to check this yourself:

new_trace_2_292.pcapng

FYI: I used a slightly modified version of your sample while working on this, so that it uses a static address on every boot, to easier find it on the BLE sniffer:

lock_5340dk_ncs292_mod.zip

Look for a device advertising using the address: C0:DE:DE:AD:DE:AF (the advertising name is still the same.

Best regards,

Edvin

[chiba]

HI,

I understand that the state changes with CONFIG_BT_CTLR_SDC_MAX_CONN_EVENT_LEN_DEFAULT.

However, the default value is set to 7500 for both NCS260 and NCS292.

- NCS292:  build\ipc_radio\zephyr\include\generated\zephyr\autoconf.h

- NCS260: build\multiprotocol_rpmsg\zephyr\include\generated\autoconf.h

Why do they behave differently?