Packet loss during ISO transmission when advertising

RE: Packet loss during ISO transmission when advertising

yuxuan.cai — Mon, 19 Aug 2024 15:06:16 GMT

Hi,

Glad to know it helps!

The fact that the controller drops outdated SDU is a design decision and thus it will most likely not be fixed.

Cheers,

Yuxuan

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Mon, 19 Aug 2024 14:36:58 GMT

Hi,

We did some test with

ret = bt_iso_chan_send(&iso_cis_chan, buf, 0);

and this is much better. We don't see any packet loss for the moment (we will continue our tests in the following days).

Could you tell me if the issue you mention with the SDU will be fixed in later version or is it normal ?

Thank you for your support.

RE: Packet loss during ISO transmission when advertising

yuxuan.cai — Mon, 19 Aug 2024 12:00:07 GMT

Hi,

Sorry that it took some time.

I've managed to reproduce the packet loss you mentioned, and after discussions with the team, I believe it has to do with how the data are provided.

In your application, the SDUs are provided to the controller with sequence numbers. When an SDU is provided too late, the controller will either drop the data or send it in the next event. An SDU can be provided too late due to the asynchronous nature of the HCI interface, or in your case, the interference from an ongoing advertiser.

To address this, you could use the "Time of Arrival" mode mentioned here by setting the sequence number to 0 when calling bt_iso_chan_send(). With this I observed no missing packets locally.

If you'd like to learn about the "Timestamp" mode, please check out the iso time sync sample.

Cheers,

Yuxuan

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Wed, 14 Aug 2024 12:53:25 GMT

Hi,

You can find enclosed a patch to apply on Zephyr (v3.6.99-ncs2) used by NCS 2.7. It's only a rework of peripheral_iso and central_iso samples (central becomes RX and peripheral becomes TX).

Once the patch is applied, you just have to build and flash these 2 samples on 2 different targets (Once flashed, the targets must be close to connect to each other since connection is based on RSSI).

devzone.nordicsemi.com/.../0001_2D00_Peripheral_2D00_TX_2D00_ISO_2D00_Central_2D00_RX_2D00_ISO.patch

There is multiple defines at the beginning of the different main.c to customize the parameters we talked about.

Let me know if you have any issue with this.

RE: Packet loss during ISO transmission when advertising

yuxuan.cai — Wed, 14 Aug 2024 07:30:23 GMT

Hi,

I was using the internal framework for quick testing, and it will take some time before I can convert it into something you could run. Would you mind sharing your codes?

Cheers,

Yuxuan

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Tue, 13 Aug 2024 15:59:25 GMT

Hi,

In the following configuration:

ACL: 70ms
SDU: 120 bytes
ISO interval: 10ms
Advertising: Every second

I tried:

RTN:10 and latency: 50ms: I still have packet loss
RTN:5 and latency: 20ms: I still have packet loss

So unfortunately, I see no improvement on my side.

Could you provide me your sample code (that allowed you to have no packet loss) to reproduce it on my end ?

Thank you

RE: Packet loss during ISO transmission when advertising

yuxuan.cai — Tue, 13 Aug 2024 14:52:18 GMT

Hi,

I have tested it a bit locally and managed to reproduce the packet loss you are seeing. It does seem to be an implmentation issue on our side. Although the root cause is not identified yet, it seems to be working when NSE > 2 and FT > 1. NSE and FT are selected by the controller with the provided RTN and max transport latency, and so far the configurations you have tried are, unfortunately, not meeting the two conditions at the same time. The parameter selection algorithm of the controller is not exposed publicly and thus there are some trial & error to do. I am sorry for the inconvinience.

You mentioned RTN=10 and max transport latency=50ms - have you tested this configuration? It had no packet loss locally for me with an advertising interval=1s, ACL connection interval=70ms, max SDU size=120bytes, ISO interval=10ms.

Or alternatively, could you try RTN=5, max transport latency=20ms? This is 48_4_1 in the quality of service configuration, and it had no packet loss for me as well.

Let us know if this is helpful to you!

Cheers,

Yuxuan

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Mon, 12 Aug 2024 09:32:06 GMT

Hi,

I tried with 60/70ms ACL connection time, I still have the same packet loss.

This is my understanding of the current problem:

Advertising is 4.5ms, so in our case this is the duration of about 3 ISO frames (3 * 1.5ms). It means that 4 retransmissions should be sufficient to cover the collisions. I don't know how much time takes ACL connection, but let's say 1.5ms (feel free to correct me), it would mean 5 retransmissions in the worst case scenario.

So, with 10 RTN and a latency of 50ms, I don't see what can cause packet loss ? I think it should cover every collision.

I don't understand why the worst case scenario (longest duration where ISO frames can't be sent because of collision) isn't known and why a configuration (retransmission and latency) is not possible to remove entirely this packet loss.

We don't have specific requirements on latency and retransmission (as far as it's reasonable), if it allows no packet loss. We are in connected mode (CIS).

Thank you.

RE: Packet loss during ISO transmission when advertising

yuxuan.cai — Mon, 12 Aug 2024 07:57:01 GMT

Hi,

The ACL connection interval you are using seems a bit too large. Could you try a smaller value of 60 or 70ms?

Due to clock drifts between devices, the peripheral will perform window widening to ensure it can receive packets from central. Simply put, the larger connection interval it is, the larger window widening it will be. This window widening on ACL connection may block the CIS packet reception.

Cheers,

Yuxuan

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Fri, 09 Aug 2024 18:02:54 GMT

Hello,

So this is my current configuration:

Advertising: 1/1.2s
ACL: 450/750 ms
RTN: 13
Latency: 100 ms

In theory, we should have no packet loss with this configuration (related to Soft Device internal mechanism/scheduling).

The result of my tests shows:

Increasing latency and RTN reduces packet loss, I now see 0.1% packet loss instead of 1%. However, it's still too much loss for a "best case scenario" (2 boards next to each others).
Increasing ACL connection is not reducing packet loss.

I'm sorry to insist, but I need a configuration with 0% packet loss in the "best case scenario". This is mandatory for our product to have the packet loss to a minimum (even though we know we can't have 0% packet loss when deployed).

To my knowledge, NRFSniffer does not support Isochronous connection. If it's not supported, I can provided you the code I run to reproduce it on your side. That would allow you to sniff the traffic on your side.

As a reminder, if I remove advertising I see absolutely no packet loss. (Even with 30/60ms ACL connection time)

RE: Packet loss during ISO transmission when advertising

yuxuan.cai — Fri, 09 Aug 2024 15:09:09 GMT

Hi,

Here is another possibility.

The ISO reception could be blocked by the ACL connection. To establish a CIS, you shall have an ACL connection by the spec. These two connections have individual intervals, and it is possible/expected that they would interfere with each other. What is the connection interval you are using for the ACL connection? It is recommended to use an ACL connection interval larger than ISO interval (say 60ms or 70ms when the ISO interval is 10ms) to reduce the inteference. Such interference might already exist before, and it was not noticed because of the retransmission. With the advertising activity of the peripheral, the radio becomes busier and thus the packet loss is noticed.

It is not guaranteed to work but could you please try a RTN = 13 with a max transport latency >= 50 ms? This allows transmitting the same ISO packet across 5 ISO intervals, in which each interval the packet could be retransmitted 3 times.

Also, it would be nice if you have any sniffer logs that we could look into.

Cheers,

Yuxuan

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Fri, 09 Aug 2024 14:30:20 GMT

Hello,

My advertising is 1 second, not 20ms.

Sorry for the confusion it was only to amplify what I didn't understand in one post. You can forget this value of 20ms.

So since my advertising is (about) every second, my diagram is valid (I think) and I don't understand why the parameters (rtn:2, latency: 20ms) do not work. Can you provide me with an example where this doesn't work ?

If possible, can you provide me with the correct parameters to have no packet loss with 1s advertising, 10ms ISO interval and 128 bytes payload ? (In my opinion, the correct parameters are predictable)

Thank you

RE: Packet loss during ISO transmission when advertising

yuxuan.cai — Fri, 09 Aug 2024 14:15:13 GMT

Hi,

Sorry for chiming in. As you already know, with an advertising interval set to 20ms, the actual advertising interval could range from 10ms to 30ms due to the random offset. So in the updated diagram, it is possible that the retransmitted Frame B is also blocked by the advertising activity, when the actual advertising interval is ~10ms. Since the retransmission number (RTN) is set to 2, and Frame B has been transmitted twice without a success, Frame B will be flushed, resulting in a packet loss.

For reference, the basic audio profile (BAP) for Bluetooth low energy defines two sets of quality of service configurations: low latency and high reliability (See 5.6.2 QoS Configurations in https://www.bluetooth.com/specifications/specs/basic-audio-profile-1-0-1/). There you can see a RTN of 2 is still in the realm of "low latency". In your application, to ensure a high reliabitliy without a high packet loss rate, I'd recommend a much higher RTN.

As a side note, the Softdevice Controller would select flush timeout (FT) and number of subevents (NSE) based on the given RTN and max transport latency. I wouldn't go into details about FT and NSE here to keep the answer short. However, the Softdevice Controller would always prioritize max transport latency over RTN, as by the Core Specification, the RTN is only a recommendation, not a mendatory requirement. Therefore, I would suggest setting a max transport latency that your application can accept first and foremost, and then increase the RTN to achieve better reliability.

Hope this helps!

Cheers,

Yuxuan

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Fri, 09 Aug 2024 11:42:54 GMT

I have updated my diagram according to your previous message:

If I follow correctly, with 2 retransmission and a latency of 20ms, we shouldn't have any loss. This is what is represented on the diagram.

So, can you explain to me, why 2 retransmission and a latency of 20ms does not solve this packet loss issue ?

Thank you

RE: Packet loss during ISO transmission when advertising

Einar Thorsrud — Fri, 09 Aug 2024 10:09:55 GMT

[quote user="thomas_hexploy"]How is this implemented in the SoftDevice ? Can you confirm that the ~10ms delay is blocking any transmission/reception ?[/quote]

No, it is not. Advertising blocks for about 4.5ms per advertising event. But due to the random offset this is not consistent so you cannot schedule the advertising events to always be between other activity. And the ISO transmissions happen at a fixed interval. So as you have one fixed interval (ISO) and one gliding and varying interval (advertising) whese will collide from time to time in a non-deterministic way.

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Fri, 09 Aug 2024 09:48:34 GMT

Hello,

Thank you for your response. However, there are two points I'd like to clarify:

I don't quite understand why these 10ms would cause a collision. From my perspective, this is only a delay and should not interfere with transmission or reception. If we consider the minimum advertising interval allowed by the specification, which is 20ms, this would imply that up to 33% (10ms random delay divided by 30ms, which is the total advertising duration, 20ms + random delay) of the bandwidth might be unavailable. Therefore, in my understanding, this delay should not cause collision with any reception or transmission.(I used the information provided in "Legacy advertising" section of https://www.bluetooth.com/blog/periodic-advertising-sync-transfer/)

How is this implemented in the SoftDevice ? Can you confirm that the ~10ms delay is blocking any transmission/reception ?

Even if we treat the 10ms delay as part of the advertising, it would suggest that advertising takes 14.5ms, which is still under 20ms (the time between 3 retransmissions). So, I don’t understand how a packet could be lost with 3 retransmissions and a 40ms latency (as I tested). The following diagram illustrates my perspective (I simplified it, focusing on lost packet):

Could you please point out where I might be mistaken?

Thank you.

RE: Packet loss during ISO transmission when advertising

Einar Thorsrud — Fri, 09 Aug 2024 07:27:08 GMT

Hi,

[quote user="thomas_hexploy"]increasing retransmission and latency improve the situation however with my configuration[/quote]

That is good.

[quote user="thomas_hexploy"]If advertising takes 4.5ms, with retransmission set to 2 and latency set to 20ms, it should be enough to cover all collision with advertising. Isn't it ?[/quote]

In principle, yes. However, legacy advertising packets have a random offset of 0-10 ms per the Bluetooth sepcification, so you cannot schedule it exactly where it would fit in between the ISO packets.

[quote user="thomas_hexploy"]

In this section it is stated that "For optimal scheduling, the periodic advertising interval and ISO interval should have a common factor, and the sum of the periodic and extended advertising timing-event lengths should be less than the BIG reserved time".

So, do you think we could reduce the impact of advertising by configuring these ?

[/quote]

This applies to periodic avertising only, but as that was not mentioned before I assumed you are using only legacy (normal) advertising. If that is the case(?), this is not relevant.

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Fri, 09 Aug 2024 06:50:19 GMT

Hi,

I tried the following configuration:

retransmission: 1, latency: 30ms or 100ms: No change
retransmissions: 3, latency: 10ms: No change
retransmissions: 3, latency: 30ms: Reduce packet loss by factor 2.5 (so packet loss when advertising is ~25% instead of 65%)

So, increasing retransmission and latency improve the situation however with my configuration I don't understand why we don't have 0 packet lost. If advertising takes 4.5ms, with retransmission set to 2 and latency set to 20ms, it should be enough to cover all collision with advertising. Isn't it ?

Moreover I found few interesting thing in the Soft Device scheduling documentation (docs.nordicsemi.com/.../scheduling.html):

In "BIS timing" (docs.nordicsemi.com/.../scheduling.html, there are few parameters that seems to transpose to CIS:

BT_CTLR_SDC_BIG_RESERVED_TIME_US: Equivalent of BT_CTLR_SDC_CIG_RESERVED_TIME_US for CIS
BT_CTLR_SDC_PERIODIC_ADV_EVENT_LEN_DEFAULT: Is it used by CIS ?

So, do you think we could reduce the impact of advertising by configuring these ?

"CONFIG_BT_CTLR_SDC_PERIODIC_ADV_EVENT_LEN_DEFAULT: The time set aside for periodic advertising each periodic advertising interval in microseconds. The event length is the primary parameter for how much data can be transmitted by the periodic advertiser without scheduling conflicts occurring.".
My understanding is that we can reduce scheduling conflict (advertising) using this parameter. If so, which value should we use ? If not, I'm not sure to understand what this value does, could you provide me with more insight ?

Thanks

RE: Packet loss during ISO transmission when advertising

Einar Thorsrud — Thu, 08 Aug 2024 11:43:50 GMT

Hi,

The numbers you are seeing make sense. The reason is that a legacy advertising event takes about 4.5 ms in our implementation (including time for a potential scan request and scan response). And an ISO event with 128 byte takes about 1 ms as you write, but including overhead it is 1.5 ms. So in this case, with an ISO packet every 10 ms, there likelyhood of a collision is above 50%, as you are seeing. In other words, what you are seeing is expected.

You may want to consider increasing the retransmission coun and transport latency to better handle the packet loss.

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Wed, 07 Aug 2024 15:55:43 GMT

Hi,

I understand what you are saying but I think the packet loss is excessive.

I used the advertising parameters 1-1.2s during 10 minutes:

Maximum possible advertising: 1 * 10 * 60 = 600 advertisements.
Number of packets lost: 393

If there is no advertising, there is no packet loss, so I can say that packet loss are caused by advertising.

It means that 393/600 = ~66% of the advertising packets causes a packet lost. (if I take the worst case scenario, meaning 500 advertisements, it would be 393/500 = ~79%)

If I try to keep it simple, with a 2M PHY (and retransmission set to 1):

We send an ISO packets every 10ms, let's say that sending an ISO frame takes 1ms (128 bytes payload), it means ISO uses 10% of the bandwidth. I have trouble understanding that with 90% of the bandwidth available, 66% of the advertising packets cause ISO frames to be lost.

Could you explain why there is so much collision ?

Increasing retransmission is not reducing the packet loss. Why ?

Thank you

FYI, I use the following parameters, with one CIS channel (extracted from Zephyr sample peripheral_iso and central_iso):

static uint16_t latency_ms = 10U; /* 10ms */
static uint32_t interval_us = 10U * USEC_PER_MSEC; /* 10 ms */

param.sca = BT_GAP_SCA_UNKNOWN;
param.packing = 0;
param.framing = 0;
param.c_to_p_latency = latency_ms; /* ms */
param.p_to_c_latency = latency_ms; /* ms */
param.c_to_p_interval = interval_us; /* us */
param.p_to_c_interval = interval_us; /* us */

RE: Packet loss during ISO transmission when advertising

Einar Thorsrud — Wed, 07 Aug 2024 14:18:26 GMT

Hi,

This is not due to a regression, but a difference in priorities in the Packetcraft controller and the SoftDevice controlloller. The Packetcraft controller always prioritizes ISO. The advantage with that is that it is less likely to get ISO packet loss. The disadvantage: other activities are influenced more. This may result in link loss, no packets being sent at all or reduced randomness in the random offsett for the advertiser.

The priority between ISO and other activities is not configurable.

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Wed, 07 Aug 2024 13:43:31 GMT

We did an other test with NCS 2.6, Packet Craft for CPU Net (ble5-ctr-rpmsg_18929.hex) and an "aggressive advertising" (30/60ms).

=> We have almost no packet loss (few packets times to times), clearly below 0.1% in the same test conditions.

So I keep thinking, this is a regression from previous NCS. Can you confirm this ? How can we fix it ?

RE: Packet loss during ISO transmission when advertising

thomas_hexploy — Tue, 06 Aug 2024 15:34:40 GMT

Hello,

We are migrating from NCS 2.1 with Packet Craft firmware (CPU NET) and we did not have this problem (at least it was not so significant).

With the configuration described in my first message, I advertised using different parameters (provided to bt_le_adv_start):

No advertising: No packet loss => 0% loss
BT_GAP_ADV_FAST_INT_MIN_1 (30 ms) and BT_GAP_ADV_FAST_INT_MAX_1 (60 ms): 568 packets lost (out of 3425) => ~17% loss
BT_GAP_ADV_FAST_INT_MIN_2 (100 ms) and BT_GAP_ADV_FAST_INT_MAX_2 (150 ms): 226 packets lost (out of 3400) => ~7% loss
BT_GAP_ADV_SLOW_INT_MIN (1 s) and BT_GAP_ADV_SLOW_INT_MAX (1.2 s): 21 packets lost (out of 2800) => ~1% loss

Looking at these numbers it seems that the advertising almost always causes an ISO packet loss.

Even though we were to use the best case scenario (advertising parameters between 1 and 1.2s), we would have a minimal 1% loss, which is too much for our use case.

With Packet Craft, we used the advertising parameters between 30 and 60ms, and we were clearly below 1% packet loss. So I don't think it's normal to have this today, and it seems more like a regression.

What do you think ?

Thanks

RE: Packet loss during ISO transmission when advertising

Einar Thorsrud — Tue, 06 Aug 2024 14:55:02 GMT

Hi,

When advertisign while in an ISO connection there will be collisions from time to time. And in this case, the stack prioritizes the advertising packets. To reduce the ISO packet loss you could increase the number of ISO retransmissions, and ansure you use an advertising interval that is as high as possible.