Inconsistent BLE data rates

[matt-lionembedded]

Hello,

I'm working on a project where we are doing the following:

• Collecting sensor data and storing it in an external SPI flash
• Retrieving that sensor data from the SPI flash, reading it out into a module global array
• Sending that sensor data from the array, in <23 byte chunks, over BLE to a phone using notifications

The condensed version of our transfer protocol (a fairly standard one) is as follows (app == phone, device == nrf52):

1. App sends message to device to begin recording data.
2. App sends message to device to indicate that it is ready to receive the next page of data, when it is ready to be sent.
3. Device collects data and when it has 1 SPI-flash-page's-worth of data, writes that data to SPI flash.
4. If the app has sent a message to the device saying that it is ready for the next page of data, read that page out of SPI flash into an array and send it in chunks over BLE to the app.
5. If the app has not yet sent a message to the device saying it is ready for the next page of data, wait until it does so, then repeat from 4.
6. Repeat from 3. 

The device always sends a page when it is available if the app has said that it is ready for it. Due to a variety of factors, sometimes the BLE data throughput is slower than the rate at which we are collecting data. In this case, "backlogged" pages are sent as soon as the app has indicated it is ready to receive another page (i.e. as soon as it has finished receiving the previous page).

Onto the interesting part:

• If, at the time the app sends the "ready for next page" command to the device, there is a backlog of pages to be sent, the page is retrieved from SPI flash immediately and sent over BLE at a rate of ~5000B/s.
• If, at the time the app sends the "ready for next page" command to the device, there is NOT a new page ready to be sent (i.e. the device is still collecting data and hasn't collected 1 pages worth yet), when the page is ready to be sent it gets read out of SPI flash and sent over BLE at a rate of ~750B/s. 

The mechanism by which the page is retrieved from SPI flash is exactly the same in both cases (the same function even), and I've performed my measurements starting at the moment AFTER the page has already been loaded from SPI flash into an array, so this doesn't have anything to do with SPI transfer rates or priorities (I don't believe). 

I've tested this both with our proprietary app and nRFConnect, on iOS and android, on multiple phones.

What I don't understand is why there is such disparity in the two scenarios described above. As I said, the code path from the time I start measuring is exactly the same for both cases. The only difference is that in the fast data rate case, there is very little time between the "I'm ready for another page" and when that page gets sent. In the slow data rate case, there is often some hundreds of milliseconds between when the "I'm ready for another page" command is received by the device and when that next page is actually ready to be sent. So far as I can tell, the negotiated connection parameters (conn interval in particular) aren't updated or changed in between the two scenarios. All else being equal, if the connection interval remained the same, and the size of the packets being sent remained the same, I'm not sure why I'm getting such inconsistent transfer rates and why they are so reliably different in the two scenarios provided. 

Is there another BLE connection setting or nrf52 configuration that might give a clue as to why this is happening? Has anyone seen something similar? Are there any other tools I can use to get to the bottom of this issue? 

Thank you

Matt

[matt-lionembedded]

Hi Dmitry,

Thanks for the clarification. I was able to update the tx_queue size via the following code:

ble_cfg_t ble_cfg;
ble_cfg.conn_cfg.conn_cfg_tag = APP_BLE_CONN_CFG_TAG;
ble_cfg.conn_cfg.params.gatts_conn_cfg.hvn_tx_queue_size = 4;
err_code = sd_ble_cfg_set(BLE_CONN_CFG_GATTS, &ble_cfg, ram_start);
APP_ERROR_CHECK(err_code);

But this had no effect on the issue at hand (i.e. still seeing the fast/slow cases). I tested this with the extended event length and the default event length with the same results. I also tried a tx_queue_size of 8. Is there something else I need to do to configure my device to use the additional tx_queue size, or is setting it in the above way sufficient?

I see the same thing you're seeing in the trace. For clarity, I was using nRFConnect app on Android for this test as the host. Is there a reason why it would respond so slowly (~75ms) in the "slow case" and more quickly (~5us) in the "fast case"? Is this something that is fixable on the device/firmware side, or are we at the mercy of the phones here? Is there a bluetooth configuration parameter that governs these response times? 

EDIT: Come to think of it, should the tx_queue_size even matter, since I'm not adding any new packets to the queue until I've already received BLE_GATTS_EVT_HVN_TX_COMPLETE?

[Dmitry]

matt-lionembedded said:

 Is there something else I need to do to configure my device to use the additional tx_queue size, or is setting it in the above way sufficient?

As you figured out yourself - you need to fill whole tx queue to have any gain from these settings.

It's hard to say why the response time slows down (I'm not android programmer, maybe there are some settings in android API). AFAIK many phones have shared radio part for BT and wifi, and OS may increase BT priority for some time after host-initiated operation, after that shrinking BT timeslots to minimum - to give maximum resources for wifi... just my thoughts, you can try to disable wifi and see what's happen.

[haakonsh]

I think Dmitry is right, either the HandleValueNotification/Indication(HVX) tx buffers are not filled when ready, or the smartphone stacks lower the priority of BLE tasks after inactivity. 

I suggest you test a few different smartphones and see how they behave, you might be testing with a phone that has a poor BLE stack. 
 
I also suggest you study how you queue your HVX tx buffers in your two cases. The ble_app_uart server example should serve as a template on how to properly queue your data. 

[matt-lionembedded]

In general I've found Dmitry's final conclusion to be true. Packet streaming after a host-intiated communication is very quick, packet streaming after a client-initiated communication is slower. I was able to verify this on a variety of phones and apps. In fact, I found that during the "slow case" of streaming (i.e. client-initiated communication prior to packet streaming), if I sent any data at all from the host to the client the transfer speed would return to the higher rate for the remainder of the data being streamed for that page. Practically speaking, the solution here was to adjust our protocol such that:

1. Client sends packet to Host indicating it is ready to send a new page

2. (This is the important one) Host sends packet to Client ACK'ing the previous message (and functionally claiming priority on the Host's radio stack for subsequent BLE activity)

3. Client streams page of data to Host

4. Repeat for every page

All in all an interesting journey into interoperability between SoCs and commercial cellphones and bluetooth connection configuration. 

Just to address the tx buffer/queue issue - This did not come in to play for me as in both fast/slow cases I was never adding more than one packet to the buffer at any given time. i.e., I am only adding a new packet once I've received the BLE_GATTS_EVT_HVN_TX_COMPLETE event, at which point the queue should be cleared.