Maximizing Notification throughput

[CktDesigner]

I'm attempting to transfer 256K bps (1 244-byte notification every ~7.5ms) using an nRF52840 using Notifications and am not getting close to the necessary bandwidth.  I'm getting the (dreaded) NRF_ERROR_RESOURCES error and have a few questions about the way that the S140 soft devices handles notifications:   I used sd_ble_gap_phy_update to specify BLE_GAP_PHY_2MBPS after receiving the BLE_GAP_EVT_CONNECTED event.    I also set BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT to 20.  My questions (so far):

1. When a characteristic value is sent using a Notification (via sd_ble_gatts_hvx), a pointer to the ble_gatts_hvx_params_t structure is sent.   In sd_ble_gatts_hvx, is this information copied, or, if BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT is greater than 1 (eg. 20) should an array of ble_gatts_hvx_params_t be created and used for this information?   In other words, if more than 1 Notification can be queued, is the information sent to sd_ble_gatts_hvx queued internally or should I maintain an array?
2. When > 1 Notifications can be queued, are the associated characteristic values always sent in the order submitted?
3. I increased the mtu size (NRF_SDH_BLE_GATT_MAX_MTU_SIZE) to 247 (as shown in several examples).    Is this a hard upper limit, or can the mtu size be increased to get additional bandwidth?
4. I also increased NRF_SDH_BLE_GAP_DATA_LENGTH to 251 (as shown in several examples).   Is this the equivalent to enabling DLE?   If not, what specifically must be done?
5. When the event BLE_GATTS_EVT_HVN_TX_COMPLETE is issued, what value is passed in the p_context argument?   Is this the pointer to the ble_gatts_hvx_params_t structure that was sent with the associated call to sd_ble_gatts_hvx that initiated the Notification?   
6. Is there any detailed documentation that I can read to help optimize the S140 with my application?

Thanks!

[Sigurd]

Hi,

1) It's copied and queued internally.

2) The packets are buffered and transmitted in the order they were queued.

3) It can be increased further, but 247 will give you the max throughput. By setting NRF_SDH_BLE_GAP_DATA_LENGTH = 251 (this is max), and NRF_SDH_BLE_GATT_MAX_MTU_SIZE = 247, then we avoid fragmentation of the ATT packet into several on-air data packets.

4) Yes.

5) No. As far as I can see, it's only a parameter to the event handler that you pass when the observer is registered. e.g. here only NULL is passed,

// Register a handler for BLE events.
NRF_SDH_BLE_OBSERVER(m_ble_observer, APP_BLE_OBSERVER_PRIO, ble_evt_handler, NULL);

6) We have a chapter about throughput in the SDS, see this link. This blog-post from novel bits might be helpful as well.

For increased throughput, I recommend increasing NRF_SDH_BLE_GAP_EVENT_LENGTH in sdk_config.h to e.g. 320, and enable connection event length extension.

You can enable Connection Event Length Extension with a function like this:

void conn_evt_len_ext_set(bool status)
{
    ret_code_t err_code;
    ble_opt_t  opt;

    memset(&opt, 0x00, sizeof(opt));
    opt.common_opt.conn_evt_ext.enable = status ? 1 : 0;

    err_code = sd_ble_opt_set(BLE_COMMON_OPT_CONN_EVT_EXT, &opt);
    APP_ERROR_CHECK(err_code);

}

Then call conn_evt_len_ext_set(true) after you have enabled the SoftDevice.

If you still are having issues with throughput, please do a sniffer trace with nRF Sniffer.

[CktDesigner]

Hi Sigurd,

I'm a bit confused on what I'm seeing.   I though that I was following the procedure outlined in sd_ble_gatts_hvx documentation: 

The application can keep track of the available queue element count for notifications by following the procedure below:

• Store initial queue element count in a variable.
• Decrement the variable, which stores the currently available queue element count, by one when a call to this function returns NRF_SUCCESS.
• Increment the variable, which stores the current available queue element count, by the count variable in BLE_GATTS_EVT_HVN_TX_COMPLETE event

My application has several sensors that can send Notifications, and I'm using FreeRTOS, so I created a small task that does nothing but collect (via a queue) notification requests from other tasks (that control the sensors) and send them.   

For testing purposes, I'm using a Samsung S8 running nRF Connect.as a Central.

I set BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT to 20 and maintain counters for packets written (notification requests), packets sent (notifications sent via sd_ble_gatts_hvx) and packet failures (errors from sd_ble_gatts_hvx - they are always NRF_ERROR_RESOURCES).   I never see more than 1 packet sent (when BLE_GATTS_EVT_HVN_TX_COMPLETE is invoked).

Here is the code for the body of the send task:

    // While there is space in the BLE queue AND there are packets to send...
    while(    (bleNotificationsAvailable > 2) 
           && (uxQueueMessagesWaiting(characteristicQueue) > 0) ) {
      // Still room in TX QUEUE and a characteristic is waiting in the queue
      // Read the queue item
      xStatus= xQueueReceive(characteristicQueue,&charToSend,0);
      if( xStatus == pdPASS ) {	// Read from queue was successful
        ble_gatts_hvx_params_t hvx_params;

        memset(&hvx_params, 0, sizeof(ble_gatts_hvx_params_t));

        size                   = charToSend.size;

        hvx_params.handle      = charToSend.charHandle;
        hvx_params.type        = BLE_GATT_HVX_NOTIFICATION;
        hvx_params.offset      = 0;
        hvx_params.p_len       = &size;
        hvx_params.p_data      = charToSend.charPtr;

        if(xStatus == pdPASS) {
          err_code= sd_ble_gatts_hvx(charToSend.connHandle,&hvx_params);
          // size will reflect the number of bytes sent
          if(    (err_code == NRF_SUCCESS)
              && (charToSend.size == size)
            ) {
            bleNotificationsAvailable--;
          } else {
            bleNotificationsFailed++;
            NRF_LOG_DEBUG("Packet Send Failure: %d",err_code);
            break;
            //APP_ERROR_CHECK(err_code);
          }
        }
      } else {
        break;
      }
    }

Here is the code for an emulated sensor that attempts to send a packet every X ms.   X needs to be about 7.5, but is currently set to 20 ms.   It sends the packet information to the send task via the characteristicQueue.

    // Send value if connected and notifying
    if (service->conn_handle != BLE_CONN_HANDLE_INVALID) {

        charQueueItem          thisChar;

        thisChar.connHandle      = service->conn_handle;
        thisChar.charHandle      = service->mic_data_handles.value_handle;
        thisChar.size            = sizeof(micDataCharacteristic);
        thisChar.charPtr         = (uint8_t *)charToSend;

        // Send the packet info
        xStatus= xQueueSendToBack(characteristicQueue, &thisChar, 0);
        if(xStatus != pdPASS ) {
          itemsInQueue= uxQueueMessagesWaiting(characteristicQueue);
          //fprintfsock("Could not send MIC packet to the queue.\n");
          while(1);
        }
        // Increment the written count
        bleNotificationsWritten++;
        // Wake the Send process
        xTaskNotify(SendTaskHandle,SEND_PACKET_QUEUED,eSetBits);

    } else {
        err_code = NRF_ERROR_INVALID_STATE;
    }

Here is the code for that handles the BLE events:

        case BLE_GATTS_EVT_HVN_TX_COMPLETE:
            bleNotificationsSent= bleNotificationsSent + p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count;
            bleNotificationsAvailable= bleNotificationsAvailable + p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count;
            //NRF_LOG_DEBUG("HVN_TX_COMPLETE: Packets: %d",p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count);
            // Wake the Send process
            xTaskNotify(SendTaskHandle,SEND_PACKET_QUEUED,eSetBits);
            break;

        case BLE_GATTS_EVT_HVC:
            bleNotificationsReceived++;
            break;

Given the way things are set up, I didn't expect to see any NRF_ERROR_RESOURCES errors since the logic should not attempt to initiate an sd_ble_gatts_hvx unless there are at least 2 empty slots available, but I do!

Eventually, the code hits the "while(1)" statement as the Queue fills and no more entries can be added.

Any thoughts on why I"m seeing NRF_ERROR_RESOURCES?   Are there other conditions that cause this other than requests exceeding BLE_GATTS_HVX_TX_QUEUE_SIZE?

Thanks!