DLE goes to 27 when using PHY_2M

RE: DLE goes to 27 when using PHY_2M

plippo — Tue, 23 Mar 2021 11:10:07 GMT

[quote userid="2121" url="~/f/nordic-q-a/71712/dle-goes-to-27-when-using-phy_2m/301277#301277"]The size of the TX buffer of the softdevice is automatically determined by the configuration of NRF_SDH_BLE_GAP_EVENT_LENGTH and the supported PHYs. [/quote]

Yes, I saw that in the documentation but in practise I've never gotten more than 4 packets (of DLE=165) before getting NRF_ERROR_RESOURCES no matter how much I give SD memory or attribute space.

[quote userid="2121" url="~/f/nordic-q-a/71712/dle-goes-to-27-when-using-phy_2m/301277#301277"]Note that the HVN_TX_COMPLETE only comes after the connection event is finished. So you may have to think about queueing the packet even before the event. [/quote]

How is this compatible with the sample code you sent? In that code the new packet is sent in the HVN_TX_COMPLETE -handler so by definition it would not make it to the same connection event.

RE: DLE goes to 27 when using PHY_2M

Hung Bui — Mon, 22 Mar 2021 16:43:20 GMT

Hi Petri,
The size of the TX buffer of the softdevice is automatically determined by the configuration of NRF_SDH_BLE_GAP_EVENT_LENGTH and the supported PHYs.

In theory the softdevice is capable of transmitting data in the full event length configured. I would strongly suggest to try to test with the throughput example we have in the SDK where we can achieve 1.3Mbps with 2Mbps PHY.

Note that it's not always correct that the smaller the interval the higher the throughput you will have. For larger packet size there will be a sweet spot that you can fit the most of the packets inside the connection event without any "idle" period.

I'm not very certain about this "the SD has already "gotten tired of waiting" . As far as know, the softdevice is capable of refilling the buffer on the go when it's transmitting its buffer. So if there is available buffer it will accept the queue.

Note that the HVN_TX_COMPLETE only comes after the connection event is finished. So you may have to think about queueing the packet even before the event.
But it's still strange to me that the buffer is not big enough to cover the transmitting of the whole registered event length. There could be a chance that it's only calculated for 1Mbps PHY not 2Mbps PHY. And that's why you would need to refill the buffer even before the HVN_TX_COMPLETE event.

RE: DLE goes to 27 when using PHY_2M

plippo — Fri, 19 Mar 2021 14:02:21 GMT

Finally got forward with this. Looks like the issue is related to FreeRTOS thread timings in relation to SD event handling. I've simplified our protocol code as much as possible and took into use semaphore protected sending (producer-consumer pattern) with event handler triggered resend (a bit like in your sample code). This refactoring resulted a worsening of transfer speed from 50kB/s to 10kB/s.

As far as I understand the situation is as follows:

Data is sent from datasource to the protocol handler for sending over BLE notification
The protocol code pushes the notification to the SD (sd_ble_gatts_hvx). The SD receives max 4 updates until it responds with NRF_ERROR_RESOURCES. Setting semaphore count to 4 protects from this happening so the sender just waits until the event comes.
SD sends BLE events (BLE_GATTS_EVT_HVN_TX_COMPLETE) via IRQ which triggers calls to nrf_sdh_evts_poll in a high priority event thread.
Event handler releases the semaphore allowing next notification to be pushed to SD.

When the PHY_2M is in use the transmission of the 4 packets of up-to 165 B takes about 3.0-4.2 ms depending on the size of the packets. By the time the HVN_TX_COMPLETE-event is handled and semaphore is triggered and the sending thread awoken to push the update into SD, the SD has already "gotten tired of waiting". SD then decides that there is no more data to send until next interval which results anything from 3-7 packets sent per interval. During the waiting the sending thread of course fills the buffer in SD so that when the new interval starts, there is again 4 ready to send.

When PHY_1M is used the 4 packet memory lasts for 5-6 ms which gives the rest of the system enough time to handle sending the new data to SD resulting more packets per interval.

I tried to add more memory to the softdevice (linker RAM start address) as well as larger ATTR_TABLE in sdk_config.h, but neither changed the situation. No matter what I did the SD only accepts 4 notifications. Is there a way to increase the buffer on the SD?

I also tried to call "nrf_sdh_evts_poll" directly in the IRQ handler, but that didn't work with FreeRTOS threads for some reason (got stuck. maybe thats why your nrf_sdh_freertos.c also uses the event thread).

Now that I know what causes the slowdown I can try to design the protocol code to respond as fast as possible, but I'm not too optimistic...

RE: DLE goes to 27 when using PHY_2M

plippo — Fri, 05 Mar 2021 13:21:50 GMT

I did some timing printouts in the event handling and the delay from IRQ to event handling was just a couple of ms, so nothing is hanging there.

I tried with 100ms connection interval and the amount per event seems to vary from 3 packets to 8 (2.5ms - 7ms, most packets were big, couple of small ones in the middle) which is the same as with interval 45 ms.

Next I set the conn interval to 7,5 (like your sample, negotiations resulted 11ms) and the situation was identical. 3-8 packets per interval but total speed was of course much faster since transmission continued much sooner (about 50% time spent waiting for next interval).

I'll continue studying this next week but it's really weird that PHY1M doesn't cause this issue...

RE: DLE goes to 27 when using PHY_2M

Hung Bui — Fri, 05 Mar 2021 10:19:47 GMT

Hi Petri,
We would need to have a look at your code to see how you call nrf_sdh_evts_poll(). At which priority level do you have your sending thread running ?

From what I know of, if you don't pull the event out from the stack event buffer, it will start NACKing the packets from the central. I suspect that there could be some share buffer here and when you don't pull the event out of the buffer, you have less buffer for the notification.

I would suggest to test on how many notification packets you can queue until you receive NRF_ERROR_RESOURCES in your case (you can use a long connection interval like 1s to make it easier to test). And compare that to what you have in my example.

You can also think of implementing the same way of calling nrf_sdh_evts_poll() in my example and see if it has any effect on the throughput.

RE: DLE goes to 27 when using PHY_2M

plippo — Fri, 05 Mar 2021 09:06:38 GMT

Hello,

I finally found the DK in our office and managed to run some tests. I did three runs with the following settings:

1. Your sample unmodified (I did have to give Softdevice 256 bytes more RAM)
2. sample with conn_interval min & max set to 45ms (to match what our settings default to when connecting to Android)
3. Conn interval of 45ms and DLE set to match ours (165)

In all cases the transfer speed was high and the packets are transferred over most of the connection interval.

I'm really running out of ideas on what could cause our weird behavior. Also a curious detail: in the 165 DLE test the DLE gets set to 165 / 165 on 3rd try or something. (initially 165 / 27 since phone wants 251 / 27).

devzone.nordicsemi.com/.../UART_5F00_Test_5F00_45ms.btt devzone.nordicsemi.com/.../UART_5F00_Test_5F00_45ms_5F00_DLE165.btt devzone.nordicsemi.com/.../UART_5F00_Test_5F00_orig.btt

EDIT: An idea occurred to me. I'm using a separate thread for handling SD events which calls nrf_sdh_evts_poll() in a loop when triggered in SD_EVT_IRQHandler. The event thread has the same priority as my sending thread, but it should get time to run during the sleep of the send thread. Assuming the event thread is not being run properly, is it possible that the softdevice stops sending data if the events are not delivered?

RE: DLE goes to 27 when using PHY_2M

Hung Bui — Tue, 23 Feb 2021 15:32:51 GMT

Hi Petri,

Could you try to reproduce the issue on a simple application so that we can have a test here ?
I also attached here a small example that you can test. It's a modified version of ble_app_uart that send 100kB of data (500 packets of 200bytes each). At 7.5ms I can see the throughput of around 100kB/s when the phone is close.
You can test by connect using the nRFToolbox app =>Uart Profile to connect, and then click button 0 on the nRF52DK. Or you can use NRF Connect, and enable CCCD before testing.

The fundamental difference I can observe here is that the MD bit at the end of the connection event is set on the last packet from the slave, meaning the slave can send more data but the central didn't want to receive more.

When in your case the slave didn't want to send more data. The example is from SDK v17.0.2 and for nRF52832 DK. Please use RTT logging when testing.

devzone.nordicsemi.com/.../ble_5F00_app_5F00_uart-_2D00_-Througput.zip

RE: DLE goes to 27 when using PHY_2M

plippo — Tue, 23 Feb 2021 12:31:33 GMT

This loop resides in a "dlServiceSend" function which is part of our communication stack. There is a thread outside that calls that code in a loop until there is no more data to send. So the next call comes in as soon as the while loop (and the function) is exited.

In my debug printouts it is clearly visible that the waiting happens in the while loop above until the connection interval is over (usually around 35 ms). So in short there is data waiting to be sent which causes it to wait in that while loop because sd_ble_gatts_hvx returns NRF_ERROR_RESOURCES.

RE: DLE goes to 27 when using PHY_2M

Hung Bui — Tue, 23 Feb 2021 10:48:23 GMT

Hi Petri,

In your code:

while (true)
{
    error = sd_ble_gatts_hvx(m_conn_handle, &hvx_params);
    if (error != NRF_ERROR_RESOURCES) break;
    threadSleepMs(1);
}

What happens when sd_ble_gatts_hvx returns NRF_SUCCESS ? From what I can see it will just break out of the while loop.
How do you queue the next packets when the first one is successfully queued ?
Please try to test with just queueing more data right away when the previous packet is successfully queued.

RE: DLE goes to 27 when using PHY_2M

plippo — Mon, 22 Feb 2021 12:16:17 GMT

Hello,

Got back to office and did the following test runs (Ellisys files as attachment). Short result: No changes in behaviour from previous tests.

- 1M1M (request 1M on rx & tx) => fast

- 2M2M (request 2M on rx & tx) => slow

- 2MxM (request 2M on tx, 1M or 2M on rx) => slow

- RetAutoOnReq: Don't ask for PHY, just if requested set rx & tx to BLE_GAP_PHY_AUTO (phone chooses 1M) => fast

- 2M2M_EL400 (request 2M on rx & tx, set NRF_SDH_BLE_GAP_EVENT_LENGTH to 400) => slow

devzone.nordicsemi.com/.../PHY2MTests.zip

RE: DLE goes to 27 when using PHY_2M

Hung Bui — Fri, 19 Feb 2021 15:18:37 GMT

Hi Petri,

Please send the 1Mbps trace.

Could you try increasing the NRF_SDH_BLE_GAP_EVENT_LENGTH ?

If you test the throughput example in the NRF5 SDK you can find that at 2M PHY the throughput can be achieved is around 1.4Mbps.

RE: DLE goes to 27 when using PHY_2M

plippo — Fri, 19 Feb 2021 12:01:17 GMT

My code reads next piece of data to send as soon as previous has been queued. The sending code calls in the loop with max 1ms delay until the packet goes thru (which trigger yet another send):

// Try to send until there is buffers available (with wait, so we don't consume CPU)

while (true)

{

error = sd_ble_gatts_hvx(m_conn_handle, &hvx_params);

if (error != NRF_ERROR_RESOURCES) break;

threadSleepMs(1);

}

The debug logging over the while look indicates that the code handles about 7-8 sends on first try and then waits for 30-40 ms until it succeeds.

Event length should be ok (and indeed it works with PHY_1M):

#define NRF_SDH_BLE_GAP_EVENT_LENGTH 36

RE: DLE goes to 27 when using PHY_2M

Hung Bui — Fri, 19 Feb 2021 10:37:15 GMT

Hi Petri,

Could you also send the 1M sniffer for comparison ?

From what I can find here in the last packet before the peers stopped transmission in a connection event is that the MD (More Data) bit is not set. This means the Peripheral doesn't want to send anything further.

There could be 2 reasons for this:

- There is no data queued. We need to take a look at how you queued your data, have you queued more data when you receive BLE_GATTS_EVT_HVN_TX_COMPLETE event ?
- Your event length configuration was too short. Could you please check NRF_SDH_BLE_GAP_EVENT_LENGTH configuration in sdk_config.h and check if you used that number in nrf_sdh_ble_default_cfg_set() ?

RE: DLE goes to 27 when using PHY_2M

plippo — Fri, 19 Feb 2021 09:24:40 GMT

Hi,

Sniffing revealed a couple of things. First, the DLE is now correct so the removal of nrf_ble_gatt module cured that issue. Unfortunately that didn't help the speed as seen in the previous message.

I've attached the screen-grabs from Ellisys which reveal the following:

Last connection update before data transfer sets the parameters as:
- Transmit window size: 1.250 ms
- Transmit window offset: 18.750 ms
- Connection Event Interval: 45.000 ms
(I could not find Event Length parameter anywhere).

The data transmit happens with correct PHY (2M) and data length (165) but for some reason the transmission only happens for ~10 ms of the whole 45 ms connection interval, the rest is spent waiting.

When having 1M PHY the transmission happens over almost all connection interval (see pic) which explains the throughput difference now that DLE issue has been resolved.

I've attached also here the raw 2M sniffer log in case your experts can use it to figure out the reason for packets not being sent:
devzone.nordicsemi.com/.../PHY_5F00_2M_5F00_DLEIssue.btt

RE: DLE goes to 27 when using PHY_2M

Hung Bui — Thu, 18 Feb 2021 09:41:07 GMT

Hi Petri,

Please provide the sniffer trace. We would need to see why the throughput dropped. As you mentioned in your last reply the max TX octets remains at 165. So why the throughput dropped ? Was it because the transmitting rate dropped ? or smaller packet was sent even though the DLE TX is 165 ?

RE: DLE goes to 27 when using PHY_2M

plippo — Thu, 18 Feb 2021 07:00:10 GMT

I'm away from the sniffer and managed to distill the case to the minimal change of calling or not calling the sd_ble_gap_phy_update()-function after receiving BLE_GAP_EVT_CONNECTED. The code calls sd_ble_gap_data_length_update() after receiving BLE_GATTC_EVT_EXCHANGE_MTU_RSP as well as BLE_GAP_EVT_PHY_UPDATE (when status == 0). The row with DataLengthSet is the one where the call to sd_ble_gap_data_length_update happens.

With PHY 2M:

00:00:14 BLE GAP: onConnect to 7A:A7:A8:35:1E:5C (Random private resolvable)
00:00:15 BLE event: 0x3A - Exchange MTU Response event
00:00:15 BLE_GATTC_EVT_EXCHANGE_MTU_RSP server_rx_mtu: 161
00:00:15 BleGapService::DataLengthSet
00:00:15 BLE event: 0x22 - BLE_GAP_EVT_PHY_UPDATE
00:00:15 BLE_GAP_EVT_PHY_UPDATE (RX:2, TX:2, status:0)
00:00:15 BleGapService::DataLengthSet
00:00:15 BLE event: 0x24 - Data length update
00:00:15 BLE_GAP_EVT_DATA_LENGTH_UPDATE: max_tx_octets: 165, max_rx_octets: 27
00:00:15 BLE event: 0x12 - Connection Parameters updated
00:00:15 BLE_GAP_EVT_CONN_PARAM_UPDATE. min_conn_interval: 6, max_conn_interval: 6, slave_latency: 0, conn_sup_timeout: 500
00:00:16 BLE event: 0x12 - Connection Parameters updated
00:00:16 BLE_GAP_EVT_CONN_PARAM_UPDATE. min_conn_interval: 36, max_conn_interval: 36, slave_latency: 0, conn_sup_timeout: 500

Same code without requesting phy_update (so the speed stays PHY_1M):

00:00:05 BLE GAP: onConnect to 42:B6:5F:10:19:CC (Random private resolvable)
00:00:05 BLE event: 0x3A - Exchange MTU Response event
00:00:05 BLE_GATTC_EVT_EXCHANGE_MTU_RSP server_rx_mtu: 161
00:00:05 BleGapService::DataLengthSet
00:00:05 BLE event: 0x24 - Data length update
00:00:05 BLE_GAP_EVT_DATA_LENGTH_UPDATE: max_tx_octets: 165, max_rx_octets: 27
00:00:06 BLE event: 0x12 - Connection Parameters updated
00:00:06 BLE_GAP_EVT_CONN_PARAM_UPDATE. min_conn_interval: 6, max_conn_interval: 6, slave_latency: 0, conn_sup_timeout: 500
00:00:06 BLE event: 0x12 - Connection Parameters updated
00:00:06 BLE_GAP_EVT_CONN_PARAM_UPDATE. min_conn_interval: 36, max_conn_interval: 36, slave_latency: 0, conn_sup_timeout: 500

In both cases I receive the event BLE_GAP_EVT_DATA_LENGTH_UPDATE (max_tx_octets: 165, max_rx_octets: 27) indicating that the DLE is setup identically but the xfer speed is half of 1M case (I'm not with the sniffer right now so I can't confirm it with it).

EDIT: I confirmed same behavior with two different phones: OnePlus 8 (Android 11) and Sony XPeria XZ Premium (Android 9) so it does not feel like a phone quirk.

RE: DLE goes to 27 when using PHY_2M

Hung Bui — Wed, 17 Feb 2021 15:12:27 GMT

Hi Petri,

You can move this thread to Private if you want to provide confidential information. Please click on "Go Private" button.

In the sniffer trace, please check for which side send the request to change the DLE size.

It could be the issue on the phone that it doesn't support long DLE size on 2Mbps PHY.