Tuning multiple Connections for optimum throughput and latency

[Marius Heil]

Hi,

we are currently in the process of optimizing our connections for better throughput. Use-case is connection based FruityMesh (https://github.com/mwaylabs/fruitymesh). For the current use-case, we want to optimize the mesh connections for optimum throughput but we still need low latency. We are only optimizing for nRF52 or upwards.

- Each node has up to 4 connections (1 peripheral, 3 central), other connections on demand e.g. to mobile devices but these are temporary and can be set up with a lower interval

- We should probably enable the 2 Mbit PHY

- Most messages are smaller than 20 byte, but there are some which are around 40-50 byte, so we would like to increase the MTU as our messag esplitting has some overhead. Does this have a negative impact if sending a lot of small messages? We probably have to enable Data length Extension (DLE) at the same time, right? Maybe enabling event length extension will also help as some nodes have 4, others less connections?

- We are mostly using WRITE_CMD and WRITE_REQ only for the handshake between two nodes.

- We probably want to keep our connectionInterval at 10ms as we want to have a low latency on networks with little traffic.

- The event length is still a little unclear to me. As far as I understand, it is the maximum event length that is going to be used for 1 connection. With a connectionInterval of 10ms and an event length of 2.5ms, I should have an equal part for all connections if I use 4 (The peripheral connection will however probably overlap the central ones). Can I simply chose a long event length of e.g. 7.5ms to allow a connection to send a lot of data if the other connections do not have a lot of data to send?

- What if I set up e.g. 4 central connections with a connection interval of 10ms and an event length of 5ms. Will the throughput be equally divided between the two overlapping connections? Or could it cause a lot of latency because e.g. I have set my connection timeout to 6 seconds. (According to the scheduling documentation of the Softdevice, the priority will change once "one connection is about to time out", not exactly sure what that means).

- My connections will be set up in a way so that 1 connection will need most of the throughput while the others will need less. But I only know that after the connections were set up, so I cannot directly specify that during the sd_ble_gap_connect call. What would be good parameters to set this up?

- I did some throughput measurements in the current state and having two devboards apart with an RSSI of -80 gave me about half the throughput than having them close to each other. Is this kind of deterioration expected? I have very little wifi/ble network interference from where I am working currently.

- Also, one last ting: This might be an implementation issue on our side, but I noticed that when testing with only two devices I sometimes get a high throughput after resetting, sometimes the throughput is less than half. Is there any good idea what might cause that, or am I just doing something wrong? Could that be because I also have a scanner and an advertiser running and the events are overlapping? Can I do anything to solve a situation like this, e.g. restarting scanning so that the event will occur after the connection role?

Thanks a lot,

Marius

[Marjeris Romero]

Hi Marius,

I am not familiar with the FruityMesh mesh implementation, but it seems you are in the process of updating to newer SDK and Softdevice for supporting nRF52 devices.

If you want to do throughput measurements and experiment with different PHY data rate, data length, ATT_MTU size, connection event extension and GAP event length settings I recommend you get your hands on 2 nRF52840 DK and try to run the ATT_MTU throughput example in our SDK, here.

I think this will give you a better picture of what to expect when porting your implementation and will give you the opportunity to run throughpust tests with different settings.

To answer some of your more general questions:

Most messages are smaller than 20 byte, but there are some which are around 40-50 byte, so we would like to increase the MTU as our messag esplitting has some overhead. Does this have a negative impact if sending a lot of small messages?

No, increasing the MTU does not have any negative impact even if the size of the message is smaller.

The event length is still a little unclear to me.

The GAP event length for a given connection is the time set aside for a given connection every connection interval. We have documentation for suggested intervals and windows based on number of connections and event length in the softdevice specification,  here. From s132 v4.0.2 (SDK 13) there is something call "Connection Event Length extentions", and when enabled the connection event will continue past the event length settings, and last until either the peer drops out of the event or there is another timing event scheduled locally (e.g. another connection, a timeslot event, or other), and the event length does not have to be changed. Maybe you could enable connection event length extention for nRF52 devices instead of updating event length settings. You can read more about Connection event length extensions here. Again I recommend you test different settings for event length and enable or disable connection event length extensions in the ATT_MTU throughput example I linked above.

I did some throughput measurements in the current state and having two devboards apart with an RSSI of -80 gave me about half the throughput than having them close to each other.

The distance between two devices should not have any impact for throughput.

Best regards,

Marjeris

 

 

 

 

[Marius Heil]

Hi,

thank you for the answer about MTU and especially the documentation on suggested intervals. i wasn't aware of that and it was very helpful. In the meantime I did tons of measurements using my own implementation and tried different values.

For my use-case, i will sometimes have 1 central connection and sometimes up to 3 and I want to share the throughput equally if possible. I will have 1 peripheral connection as well most of the time. As the peripheral connection runs asynchronously, I do not need to put too much throught in it.

When I have 1 central connection, i want to have the maximum throughput available, but if I have 3, I want to equally share the throughput. I do not know during connection setup how many connections I will have, so I cannot use different event lengths.

My tests have shown me, that using a connectionInterval of 7.5ms and an event length of 7.5ms for each connection will give me the best results. The throughput is very high when only having one connection and is split into mostly equal parts when having 3 connections. I suppose the scheduler will switch the connection each time. So with 3 central and 1 peripheral connection, it will result in a connection interval of 4*7.5ms = 30ms.

This however is completely against the guidelines. According to these, i should use 2.5ms event length and 7.5ms connInterval when having three central connections. Can you please tell me if the scheduler is doing what I am thinking? Will I experiance connection drops due to this setup or is this a good setting?

Regarding the throughput, i have now observed a drop of about 10-20% with and RSSI of around -80/-85 and a drop of around 70% if the RSSI is around -90.

Marius

PS: One more thing. When using this setting and updating the PHY to 2Mbit instead of 1Mbit, I got a significant reduction of the throughput for unexplicable reasons. With all eventLengths adding up to the connInterval, the throughput increased???

[Marjeris Romero]

Hi Marius,

Marius Heil said:

I suppose the scheduler will switch the connection each time.

Sorry for the very late reply here. for some reason this ticket had disappear in the queue with other tickets... Ideally you will go into a situation with round-robin between the links, but the slave link might be placed on top of two central links, so you might get worse behaviour than that from time to time, so you should test this solution over time to see if is good enough for you.

I am going on vacation so I have asked one of my colleagues if he has anything to add to my answer, so you may hear from him soon.

Best regards,

Marjeris

[tesc]

Hi,

While I did not have much to add initially, I revisited this now and I find there to  be a few loose threads.

If you have no connections as peripheral and N connections as central, it should be optimal to schedule the event lengths to be 1/N times the connection interval. The issue arises once you put a connection as peripheral into the mix, as that peripheral connection will be randomly inserted (as it is decided by the central.) As if that is not enough, the peripheral connection will also drift relative to the central connections. Also, each connection as peripheral will most likely collide with _two_ central connection slots every connection interval, as it is unlikely to get positioned 100 % overlapping one and only one central connection event.

One thing you can experiment with for the peripheral connection is slave latency. As peripheral, the SoftDevice will skip connection events for "slave_latency" number of events if there is no data. That way you automatically shift throughput over to other connections in situations where you pack more events into the connection interval than what will fit. The drawback is increased latency "in the other direction", that is if the central has data to send.

Marius Heil said:

PS: One more thing. When using this setting and updating the PHY to 2Mbit instead of 1Mbit, I got a significant reduction of the throughput for unexplicable reasons. With all eventLengths adding up to the connInterval, the throughput increased???

I am afraid I did not quite understand your last remarks there. Please elaborate if you want us to look further into it.

There is a trade-off between bitrate and range, so what you see may be related to that. Other things to look out for is if you increase MTU size, increase event lengths (and connection interval), etc., then you increase the potential damage to throughput from interference or poor reception. The reason is simply that once you get an error in a packet, the full packet is lost. Related, although not necessarily relevant, is that long range in some situations gives way more packet loss than 1Mbps, because the bitrate is lower, leading to longer packets, increasing the chance of packet collisions.

Regards,
Terje