Maximizing Notification throughput

[CktDesigner]

I'm attempting to transfer 256K bps (1 244-byte notification every ~7.5ms) using an nRF52840 using Notifications and am not getting close to the necessary bandwidth.  I'm getting the (dreaded) NRF_ERROR_RESOURCES error and have a few questions about the way that the S140 soft devices handles notifications:   I used sd_ble_gap_phy_update to specify BLE_GAP_PHY_2MBPS after receiving the BLE_GAP_EVT_CONNECTED event.    I also set BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT to 20.  My questions (so far):

1. When a characteristic value is sent using a Notification (via sd_ble_gatts_hvx), a pointer to the ble_gatts_hvx_params_t structure is sent.   In sd_ble_gatts_hvx, is this information copied, or, if BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT is greater than 1 (eg. 20) should an array of ble_gatts_hvx_params_t be created and used for this information?   In other words, if more than 1 Notification can be queued, is the information sent to sd_ble_gatts_hvx queued internally or should I maintain an array?
2. When > 1 Notifications can be queued, are the associated characteristic values always sent in the order submitted?
3. I increased the mtu size (NRF_SDH_BLE_GATT_MAX_MTU_SIZE) to 247 (as shown in several examples).    Is this a hard upper limit, or can the mtu size be increased to get additional bandwidth?
4. I also increased NRF_SDH_BLE_GAP_DATA_LENGTH to 251 (as shown in several examples).   Is this the equivalent to enabling DLE?   If not, what specifically must be done?
5. When the event BLE_GATTS_EVT_HVN_TX_COMPLETE is issued, what value is passed in the p_context argument?   Is this the pointer to the ble_gatts_hvx_params_t structure that was sent with the associated call to sd_ble_gatts_hvx that initiated the Notification?   
6. Is there any detailed documentation that I can read to help optimize the S140 with my application?

Thanks!

[Sigurd]

Hi,

1) It's copied and queued internally.

2) The packets are buffered and transmitted in the order they were queued.

3) It can be increased further, but 247 will give you the max throughput. By setting NRF_SDH_BLE_GAP_DATA_LENGTH = 251 (this is max), and NRF_SDH_BLE_GATT_MAX_MTU_SIZE = 247, then we avoid fragmentation of the ATT packet into several on-air data packets.

4) Yes.

5) No. As far as I can see, it's only a parameter to the event handler that you pass when the observer is registered. e.g. here only NULL is passed,

// Register a handler for BLE events.
NRF_SDH_BLE_OBSERVER(m_ble_observer, APP_BLE_OBSERVER_PRIO, ble_evt_handler, NULL);

6) We have a chapter about throughput in the SDS, see this link. This blog-post from novel bits might be helpful as well.

For increased throughput, I recommend increasing NRF_SDH_BLE_GAP_EVENT_LENGTH in sdk_config.h to e.g. 320, and enable connection event length extension.

You can enable Connection Event Length Extension with a function like this:

void conn_evt_len_ext_set(bool status)
{
    ret_code_t err_code;
    ble_opt_t  opt;

    memset(&opt, 0x00, sizeof(opt));
    opt.common_opt.conn_evt_ext.enable = status ? 1 : 0;

    err_code = sd_ble_opt_set(BLE_COMMON_OPT_CONN_EVT_EXT, &opt);
    APP_ERROR_CHECK(err_code);

}

Then call conn_evt_len_ext_set(true) after you have enabled the SoftDevice.

If you still are having issues with throughput, please do a sniffer trace with nRF Sniffer.

[CktDesigner]

Hi Sigurd,

Here is a summary of where I'm at:

The following defines have been modified in sdk_config.h

#define NRF_SDH_BLE_GAP_DATA_LENGTH 251

#define NRF_SDH_BLE_GAP_EVENT_LENGTH 320

#define NRF_SDH_BLE_GATT_MAX_MTU_SIZE 247

I call:

void connEvtLenExtSet(bool onOff) {
  ret_code_t err_code;
  ble_opt_t opt;

  memset(&opt,0,sizeof(opt));
  opt.common_opt.conn_evt_ext.enable= onOff ? 1 : 0;

  err_code= sd_ble_opt_set(BLE_COMMON_OPT_CONN_EVT_EXT, &opt);
  APP_ERROR_CHECK(err_code);
}

after gap and gatt are initialized:

    // Configure and initialize the BLE stack.
    ble_stack_init();

    // Initialize modules.
    timers_init();
    buttons_leds_init(&erase_bonds);
    gap_params_init();
    gatt_init();
    //dataLenExtSet();
    connEvtLenExtSet(true);
    advertising_init();
    services_init();
    sensor_simulator_init();
    conn_params_init();
    peer_manager_init();

    // Initialize the Queues
    bleNotificationsWritten= 0;
    bleNotificationsSent= 0;
    bleNotificationsFailed= 0;
    bleNotificationsReceived= 0;
    bleNotificationsAvailable= BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT;

I update the Phy to 2MBPS using:

void updatePhy(uint16_t handle) {
  ret_code_t err_code;
  ble_gap_phys_t phys = {
    //.tx_phys = BLE_GAP_PHY_2MBPS | BLE_GAP_PHY_1MBPS,
    //.rx_phys = BLE_GAP_PHY_2MBPS | BLE_GAP_PHY_1MBPS

    .tx_phys = BLE_GAP_PHY_2MBPS,
    .rx_phys = BLE_GAP_PHY_2MBPS
  };
  NRF_LOG_DEBUG("PHY update set.");

  err_code= sd_ble_gap_phy_update(handle, &phys);
  APP_ERROR_CHECK(err_code);
}

This is called after the connection is established:

        case BLE_GAP_EVT_CONNECTED:
            NRF_LOG_INFO("Connected");
            err_code = bsp_indication_set(BSP_INDICATE_CONNECTED);
            APP_ERROR_CHECK(err_code);
            m_conn_handle = p_ble_evt->evt.gap_evt.conn_handle;
            err_code = nrf_ble_qwr_conn_handle_assign(&m_qwr, m_conn_handle);
            APP_ERROR_CHECK(err_code);
            updatePhy(m_conn_handle);
            break;

I keep track of the number of notifications being processed using the variable "bleNotificationsAvailable".   It is initialized to BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT (which is set to 20).

This variable is incremented by the number of notifications sent in the BLE callback:

        case BLE_GATTS_EVT_HVN_TX_COMPLETE:
            bleNotificationsSent= bleNotificationsSent + p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count;
            bleNotificationsAvailable= bleNotificationsAvailable + p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count;
            NRF_LOG_DEBUG("HVN_TX_COMPLETE: Packets: %d",p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count);
            // Wake the Send process
            xTaskNotify(SendTaskHandle,SEND_PACKET_QUEUED,eSetBits);
            break;

        case BLE_GATTS_EVT_HVC:
            bleNotificationsReceived++;
            break;

and decremented when a notification is sent.

Since Notifications can be sent by multiple sensors,  the "send" code is consolidated in one FreeRTOS task.  This is the body of that task:

for (;;) {
    // Block until something appears in the queue...
    // Only bit that should cause Notification is SEND_PACKET_QUEUED, so don't even check...
    xTaskNotifyWait(0x00,             // Don't clear any bits on entry
                    ~0x00,            // Clear all bits on exit
                    &bleSendNotifyBits,
                    portMAX_DELAY);   // Block indefinitely
    
    // While there is space in the BLE queue AND there are packets to send...
    while(    (bleNotificationsAvailable > 2) 
           && (uxQueueMessagesWaiting(characteristicQueue) > 0) ) {
      // Still room in TX QUEUE and a characteristic is waiting in the queue
      // Read the queue item
      xStatus= xQueueReceive(characteristicQueue,&charToSend,0);
      if( xStatus == pdPASS ) {	// Read from queue was successful
        ble_gatts_hvx_params_t hvx_params;

        memset(&hvx_params, 0, sizeof(ble_gatts_hvx_params_t));

        size                   = charToSend.size;

        hvx_params.handle      = charToSend.charHandle;
        hvx_params.type        = BLE_GATT_HVX_NOTIFICATION;
        hvx_params.offset      = 0;
        hvx_params.p_len       = &size;
        hvx_params.p_data      = charToSend.charPtr;

        err_code= sd_ble_gatts_hvx(charToSend.connHandle,&hvx_params);
        // size will reflect the number of bytes sent
        if(    (err_code == NRF_SUCCESS)
            && (charToSend.size == size)
          ) {
          bleNotificationsAvailable--;
        } else {
          bleNotificationsFailed++;
          NRF_LOG_DEBUG("Packet Send Failure: %d : %d",err_code,bleNotificationsAvailable);
          break;
          //APP_ERROR_CHECK(err_code);
        }
      } else {
        break;
      }
    }
  }

This task sends Notifications as long as there is TX queue space available and as long as a notification is present in the queue (from other tasks that handle sensors).    I thought using this code would eliminate getting NRF_ERROR_RESOURCES errors, but I still get them.    There aren't as many when I use shorter connect intervals (7.5).   Here is the code where this is set:

#define MIN_CONN_INTERVAL                   MSEC_TO_UNITS(7.5, UNIT_1_25_MS)        /**< Minimum acceptable connection interval (0.4 seconds). */
#define MAX_CONN_INTERVAL                   MSEC_TO_UNITS(650, UNIT_1_25_MS)        /**< Maximum acceptable connection interval (0.65 second). */

I thought perhaps there was some sort of race condition, so increased the comparison (as shown above) for the number of free TX queue entries to 2.    So when I activate the "Send" Task, there should be at least 2 free TX queue slots or no call to sd_ble_gatts_hvn should be made.    

However, I still get a lot of debug print statements indicating packet failures with NRF_ERROR_RESOURCES and supposedly 16 empty TX queue slots.

For completion, here is the code for a testing task that sends packets (about every 7.5 ms).  The Notifications are about 244 bytes in size:

    // Send value if connected and notifying
    if (service->conn_handle != BLE_CONN_HANDLE_INVALID) {

        charQueueItem          thisChar;

        thisChar.connHandle      = service->conn_handle;
        thisChar.charHandle      = service->mic_data_handles.value_handle;
        thisChar.size            = sizeof(micDataCharacteristic);
        thisChar.charPtr         = (uint8_t *)charToSend;

        // Send the packet info
        xStatus= xQueueSendToBack(characteristicQueue, &thisChar, 0);
        if(xStatus != pdPASS ) {
          itemsInQueue= uxQueueMessagesWaiting(characteristicQueue);
          //fprintfsock("Could not send MIC packet to the queue.\n");
          while(1);
        }
        // Increment the written count
        bleNotificationsWritten++;
        // Wake the Send process
        xTaskNotify(SendTaskHandle,SEND_PACKET_QUEUED,eSetBits);

    } else {
        err_code = NRF_ERROR_INVALID_STATE;
    }

This creates a FreeRTOS queue item that includes information about the connection handle, the characteristic handle, a pointer to the characteristic value to be sent, etc. and sends it to the queue.   It also wakes the Send task to actually send the notification if conditions permit.     If the notification can't be sent, the queue item won't be consumed.   When a BLE_GATTS_EVT_HVN_TX_COMPLETE arrives, the bleNotificationsAvailable variable is updated and the Send Task is notified to check if anything should be sent.

I didn't expect to see the NRF_ERROR_RESOURCES.   Are there any other conditions (besides the TX QUEUE being overrun) that I should consider?   My variable indicates that there are 16 slots available when the error occurs.

I am attempting to receive the notifications on a Samsung S8 running Android 9.   I have used LightBlue, nRF Connect, and my own Android app as receiving apps (central) and all do pretty much the same thing. 

What am I doing wrong???

Thanks!