Maximizing Notification throughput

[CktDesigner]

I'm attempting to transfer 256K bps (1 244-byte notification every ~7.5ms) using an nRF52840 using Notifications and am not getting close to the necessary bandwidth.  I'm getting the (dreaded) NRF_ERROR_RESOURCES error and have a few questions about the way that the S140 soft devices handles notifications:   I used sd_ble_gap_phy_update to specify BLE_GAP_PHY_2MBPS after receiving the BLE_GAP_EVT_CONNECTED event.    I also set BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT to 20.  My questions (so far):

1. When a characteristic value is sent using a Notification (via sd_ble_gatts_hvx), a pointer to the ble_gatts_hvx_params_t structure is sent.   In sd_ble_gatts_hvx, is this information copied, or, if BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT is greater than 1 (eg. 20) should an array of ble_gatts_hvx_params_t be created and used for this information?   In other words, if more than 1 Notification can be queued, is the information sent to sd_ble_gatts_hvx queued internally or should I maintain an array?
2. When > 1 Notifications can be queued, are the associated characteristic values always sent in the order submitted?
3. I increased the mtu size (NRF_SDH_BLE_GATT_MAX_MTU_SIZE) to 247 (as shown in several examples).    Is this a hard upper limit, or can the mtu size be increased to get additional bandwidth?
4. I also increased NRF_SDH_BLE_GAP_DATA_LENGTH to 251 (as shown in several examples).   Is this the equivalent to enabling DLE?   If not, what specifically must be done?
5. When the event BLE_GATTS_EVT_HVN_TX_COMPLETE is issued, what value is passed in the p_context argument?   Is this the pointer to the ble_gatts_hvx_params_t structure that was sent with the associated call to sd_ble_gatts_hvx that initiated the Notification?   
6. Is there any detailed documentation that I can read to help optimize the S140 with my application?

Thanks!

[Sigurd]

Hi,

1) It's copied and queued internally.

2) The packets are buffered and transmitted in the order they were queued.

3) It can be increased further, but 247 will give you the max throughput. By setting NRF_SDH_BLE_GAP_DATA_LENGTH = 251 (this is max), and NRF_SDH_BLE_GATT_MAX_MTU_SIZE = 247, then we avoid fragmentation of the ATT packet into several on-air data packets.

4) Yes.

5) No. As far as I can see, it's only a parameter to the event handler that you pass when the observer is registered. e.g. here only NULL is passed,

// Register a handler for BLE events.
NRF_SDH_BLE_OBSERVER(m_ble_observer, APP_BLE_OBSERVER_PRIO, ble_evt_handler, NULL);

6) We have a chapter about throughput in the SDS, see this link. This blog-post from novel bits might be helpful as well.

For increased throughput, I recommend increasing NRF_SDH_BLE_GAP_EVENT_LENGTH in sdk_config.h to e.g. 320, and enable connection event length extension.

You can enable Connection Event Length Extension with a function like this:

void conn_evt_len_ext_set(bool status)
{
    ret_code_t err_code;
    ble_opt_t  opt;

    memset(&opt, 0x00, sizeof(opt));
    opt.common_opt.conn_evt_ext.enable = status ? 1 : 0;

    err_code = sd_ble_opt_set(BLE_COMMON_OPT_CONN_EVT_EXT, &opt);
    APP_ERROR_CHECK(err_code);

}

Then call conn_evt_len_ext_set(true) after you have enabled the SoftDevice.

If you still are having issues with throughput, please do a sniffer trace with nRF Sniffer.

[CktDesigner]

OK, thanks!    I think I've done all that you suggested, but still having problems.   I'll have to get another development board to try the nRF Sniffer.     I also need to try to better manage the Notification send requests.

[CktDesigner]

Hi Sigurd,

I'm a bit confused on what I'm seeing.   I though that I was following the procedure outlined in sd_ble_gatts_hvx documentation: 

The application can keep track of the available queue element count for notifications by following the procedure below:

• Store initial queue element count in a variable.
• Decrement the variable, which stores the currently available queue element count, by one when a call to this function returns NRF_SUCCESS.
• Increment the variable, which stores the current available queue element count, by the count variable in BLE_GATTS_EVT_HVN_TX_COMPLETE event

My application has several sensors that can send Notifications, and I'm using FreeRTOS, so I created a small task that does nothing but collect (via a queue) notification requests from other tasks (that control the sensors) and send them.   

For testing purposes, I'm using a Samsung S8 running nRF Connect.as a Central.

I set BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT to 20 and maintain counters for packets written (notification requests), packets sent (notifications sent via sd_ble_gatts_hvx) and packet failures (errors from sd_ble_gatts_hvx - they are always NRF_ERROR_RESOURCES).   I never see more than 1 packet sent (when BLE_GATTS_EVT_HVN_TX_COMPLETE is invoked).

Here is the code for the body of the send task:

    // While there is space in the BLE queue AND there are packets to send...
    while(    (bleNotificationsAvailable > 2) 
           && (uxQueueMessagesWaiting(characteristicQueue) > 0) ) {
      // Still room in TX QUEUE and a characteristic is waiting in the queue
      // Read the queue item
      xStatus= xQueueReceive(characteristicQueue,&charToSend,0);
      if( xStatus == pdPASS ) {	// Read from queue was successful
        ble_gatts_hvx_params_t hvx_params;

        memset(&hvx_params, 0, sizeof(ble_gatts_hvx_params_t));

        size                   = charToSend.size;

        hvx_params.handle      = charToSend.charHandle;
        hvx_params.type        = BLE_GATT_HVX_NOTIFICATION;
        hvx_params.offset      = 0;
        hvx_params.p_len       = &size;
        hvx_params.p_data      = charToSend.charPtr;

        if(xStatus == pdPASS) {
          err_code= sd_ble_gatts_hvx(charToSend.connHandle,&hvx_params);
          // size will reflect the number of bytes sent
          if(    (err_code == NRF_SUCCESS)
              && (charToSend.size == size)
            ) {
            bleNotificationsAvailable--;
          } else {
            bleNotificationsFailed++;
            NRF_LOG_DEBUG("Packet Send Failure: %d",err_code);
            break;
            //APP_ERROR_CHECK(err_code);
          }
        }
      } else {
        break;
      }
    }

Here is the code for an emulated sensor that attempts to send a packet every X ms.   X needs to be about 7.5, but is currently set to 20 ms.   It sends the packet information to the send task via the characteristicQueue.

    // Send value if connected and notifying
    if (service->conn_handle != BLE_CONN_HANDLE_INVALID) {

        charQueueItem          thisChar;

        thisChar.connHandle      = service->conn_handle;
        thisChar.charHandle      = service->mic_data_handles.value_handle;
        thisChar.size            = sizeof(micDataCharacteristic);
        thisChar.charPtr         = (uint8_t *)charToSend;

        // Send the packet info
        xStatus= xQueueSendToBack(characteristicQueue, &thisChar, 0);
        if(xStatus != pdPASS ) {
          itemsInQueue= uxQueueMessagesWaiting(characteristicQueue);
          //fprintfsock("Could not send MIC packet to the queue.\n");
          while(1);
        }
        // Increment the written count
        bleNotificationsWritten++;
        // Wake the Send process
        xTaskNotify(SendTaskHandle,SEND_PACKET_QUEUED,eSetBits);

    } else {
        err_code = NRF_ERROR_INVALID_STATE;
    }

Here is the code for that handles the BLE events:

        case BLE_GATTS_EVT_HVN_TX_COMPLETE:
            bleNotificationsSent= bleNotificationsSent + p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count;
            bleNotificationsAvailable= bleNotificationsAvailable + p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count;
            //NRF_LOG_DEBUG("HVN_TX_COMPLETE: Packets: %d",p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count);
            // Wake the Send process
            xTaskNotify(SendTaskHandle,SEND_PACKET_QUEUED,eSetBits);
            break;

        case BLE_GATTS_EVT_HVC:
            bleNotificationsReceived++;
            break;

Given the way things are set up, I didn't expect to see any NRF_ERROR_RESOURCES errors since the logic should not attempt to initiate an sd_ble_gatts_hvx unless there are at least 2 empty slots available, but I do!

Eventually, the code hits the "while(1)" statement as the Queue fills and no more entries can be added.

Any thoughts on why I"m seeing NRF_ERROR_RESOURCES?   Are there other conditions that cause this other than requests exceeding BLE_GATTS_HVX_TX_QUEUE_SIZE?

Thanks!

[CktDesigner]

Another update:

1.   I had increased the TX_QUEUE size by modifying BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT to 20 in ble_gatts.h.    I thought perhaps doing it dynamically might have a better outcome, so I changed the #define back to 1 and used the code segment:  

    // Increase the TX QUEUE size
    ble_cfg_t ble_cfg;
    memset(&ble_cfg, 0, sizeof ble_cfg);
    ble_cfg.conn_cfg.conn_cfg_tag = APP_BLE_CONN_CFG_TAG;
    ble_cfg.conn_cfg.params.gatts_conn_cfg.hvn_tx_queue_size = 20;
    err_code = sd_ble_cfg_set(BLE_CONN_CFG_GATTS, &ble_cfg, ram_start);
    APP_ERROR_CHECK(err_code);

to attempt to increase it.    This resulted in an error that there was not enough memory, so I switched back to setting the QUEUE size to 20 in the .h file.

2. Using the code in the earlier post, I attempted to run with my emulator sending one 244 byte Notification every 7.6 ms.   Many of the notifications don't get sent because sd_ble_gatts_hvx returns error 19 (NRF_ERROR_RESOURCES) even though the variable I maintain that tracks the number of TX queue entries shows there should be 16 empty queue positions (per an NRF_LOG_DEBUG message that logs the err_code and bleNotificationsAvailable variable.)

3. I'm using a Samsung S8 phone (BLE 5.x) to receive (as the central) and set the phy to 2MBPS in the peripheral when the BLE_GAP_EVT_CONNECTED is received.    With the MIN_CONN_INTERVAL set to 7.5 ms, a significant number of packets don't get sent.    I increased the MIN_CONN_INTERVAL to 50 ms and even more packets don't get set.

Are there other resources that must be increased when the TX QUEUE size is increased?

Thanks...

[CktDesigner]

Hi Sigurd,

Here is a summary of where I'm at:

The following defines have been modified in sdk_config.h

#define NRF_SDH_BLE_GAP_DATA_LENGTH 251

#define NRF_SDH_BLE_GAP_EVENT_LENGTH 320

#define NRF_SDH_BLE_GATT_MAX_MTU_SIZE 247

I call:

void connEvtLenExtSet(bool onOff) {
  ret_code_t err_code;
  ble_opt_t opt;

  memset(&opt,0,sizeof(opt));
  opt.common_opt.conn_evt_ext.enable= onOff ? 1 : 0;

  err_code= sd_ble_opt_set(BLE_COMMON_OPT_CONN_EVT_EXT, &opt);
  APP_ERROR_CHECK(err_code);
}

after gap and gatt are initialized:

    // Configure and initialize the BLE stack.
    ble_stack_init();

    // Initialize modules.
    timers_init();
    buttons_leds_init(&erase_bonds);
    gap_params_init();
    gatt_init();
    //dataLenExtSet();
    connEvtLenExtSet(true);
    advertising_init();
    services_init();
    sensor_simulator_init();
    conn_params_init();
    peer_manager_init();

    // Initialize the Queues
    bleNotificationsWritten= 0;
    bleNotificationsSent= 0;
    bleNotificationsFailed= 0;
    bleNotificationsReceived= 0;
    bleNotificationsAvailable= BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT;

I update the Phy to 2MBPS using:

void updatePhy(uint16_t handle) {
  ret_code_t err_code;
  ble_gap_phys_t phys = {
    //.tx_phys = BLE_GAP_PHY_2MBPS | BLE_GAP_PHY_1MBPS,
    //.rx_phys = BLE_GAP_PHY_2MBPS | BLE_GAP_PHY_1MBPS

    .tx_phys = BLE_GAP_PHY_2MBPS,
    .rx_phys = BLE_GAP_PHY_2MBPS
  };
  NRF_LOG_DEBUG("PHY update set.");

  err_code= sd_ble_gap_phy_update(handle, &phys);
  APP_ERROR_CHECK(err_code);
}

This is called after the connection is established:

        case BLE_GAP_EVT_CONNECTED:
            NRF_LOG_INFO("Connected");
            err_code = bsp_indication_set(BSP_INDICATE_CONNECTED);
            APP_ERROR_CHECK(err_code);
            m_conn_handle = p_ble_evt->evt.gap_evt.conn_handle;
            err_code = nrf_ble_qwr_conn_handle_assign(&m_qwr, m_conn_handle);
            APP_ERROR_CHECK(err_code);
            updatePhy(m_conn_handle);
            break;

I keep track of the number of notifications being processed using the variable "bleNotificationsAvailable".   It is initialized to BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT (which is set to 20).

This variable is incremented by the number of notifications sent in the BLE callback:

        case BLE_GATTS_EVT_HVN_TX_COMPLETE:
            bleNotificationsSent= bleNotificationsSent + p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count;
            bleNotificationsAvailable= bleNotificationsAvailable + p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count;
            NRF_LOG_DEBUG("HVN_TX_COMPLETE: Packets: %d",p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count);
            // Wake the Send process
            xTaskNotify(SendTaskHandle,SEND_PACKET_QUEUED,eSetBits);
            break;

        case BLE_GATTS_EVT_HVC:
            bleNotificationsReceived++;
            break;

and decremented when a notification is sent.

Since Notifications can be sent by multiple sensors,  the "send" code is consolidated in one FreeRTOS task.  This is the body of that task:

for (;;) {
    // Block until something appears in the queue...
    // Only bit that should cause Notification is SEND_PACKET_QUEUED, so don't even check...
    xTaskNotifyWait(0x00,             // Don't clear any bits on entry
                    ~0x00,            // Clear all bits on exit
                    &bleSendNotifyBits,
                    portMAX_DELAY);   // Block indefinitely
    
    // While there is space in the BLE queue AND there are packets to send...
    while(    (bleNotificationsAvailable > 2) 
           && (uxQueueMessagesWaiting(characteristicQueue) > 0) ) {
      // Still room in TX QUEUE and a characteristic is waiting in the queue
      // Read the queue item
      xStatus= xQueueReceive(characteristicQueue,&charToSend,0);
      if( xStatus == pdPASS ) {	// Read from queue was successful
        ble_gatts_hvx_params_t hvx_params;

        memset(&hvx_params, 0, sizeof(ble_gatts_hvx_params_t));

        size                   = charToSend.size;

        hvx_params.handle      = charToSend.charHandle;
        hvx_params.type        = BLE_GATT_HVX_NOTIFICATION;
        hvx_params.offset      = 0;
        hvx_params.p_len       = &size;
        hvx_params.p_data      = charToSend.charPtr;

        err_code= sd_ble_gatts_hvx(charToSend.connHandle,&hvx_params);
        // size will reflect the number of bytes sent
        if(    (err_code == NRF_SUCCESS)
            && (charToSend.size == size)
          ) {
          bleNotificationsAvailable--;
        } else {
          bleNotificationsFailed++;
          NRF_LOG_DEBUG("Packet Send Failure: %d : %d",err_code,bleNotificationsAvailable);
          break;
          //APP_ERROR_CHECK(err_code);
        }
      } else {
        break;
      }
    }
  }

This task sends Notifications as long as there is TX queue space available and as long as a notification is present in the queue (from other tasks that handle sensors).    I thought using this code would eliminate getting NRF_ERROR_RESOURCES errors, but I still get them.    There aren't as many when I use shorter connect intervals (7.5).   Here is the code where this is set:

#define MIN_CONN_INTERVAL                   MSEC_TO_UNITS(7.5, UNIT_1_25_MS)        /**< Minimum acceptable connection interval (0.4 seconds). */
#define MAX_CONN_INTERVAL                   MSEC_TO_UNITS(650, UNIT_1_25_MS)        /**< Maximum acceptable connection interval (0.65 second). */

I thought perhaps there was some sort of race condition, so increased the comparison (as shown above) for the number of free TX queue entries to 2.    So when I activate the "Send" Task, there should be at least 2 free TX queue slots or no call to sd_ble_gatts_hvn should be made.    

However, I still get a lot of debug print statements indicating packet failures with NRF_ERROR_RESOURCES and supposedly 16 empty TX queue slots.

For completion, here is the code for a testing task that sends packets (about every 7.5 ms).  The Notifications are about 244 bytes in size:

    // Send value if connected and notifying
    if (service->conn_handle != BLE_CONN_HANDLE_INVALID) {

        charQueueItem          thisChar;

        thisChar.connHandle      = service->conn_handle;
        thisChar.charHandle      = service->mic_data_handles.value_handle;
        thisChar.size            = sizeof(micDataCharacteristic);
        thisChar.charPtr         = (uint8_t *)charToSend;

        // Send the packet info
        xStatus= xQueueSendToBack(characteristicQueue, &thisChar, 0);
        if(xStatus != pdPASS ) {
          itemsInQueue= uxQueueMessagesWaiting(characteristicQueue);
          //fprintfsock("Could not send MIC packet to the queue.\n");
          while(1);
        }
        // Increment the written count
        bleNotificationsWritten++;
        // Wake the Send process
        xTaskNotify(SendTaskHandle,SEND_PACKET_QUEUED,eSetBits);

    } else {
        err_code = NRF_ERROR_INVALID_STATE;
    }

This creates a FreeRTOS queue item that includes information about the connection handle, the characteristic handle, a pointer to the characteristic value to be sent, etc. and sends it to the queue.   It also wakes the Send task to actually send the notification if conditions permit.     If the notification can't be sent, the queue item won't be consumed.   When a BLE_GATTS_EVT_HVN_TX_COMPLETE arrives, the bleNotificationsAvailable variable is updated and the Send Task is notified to check if anything should be sent.

I didn't expect to see the NRF_ERROR_RESOURCES.   Are there any other conditions (besides the TX QUEUE being overrun) that I should consider?   My variable indicates that there are 16 slots available when the error occurs.

I am attempting to receive the notifications on a Samsung S8 running Android 9.   I have used LightBlue, nRF Connect, and my own Android app as receiving apps (central) and all do pretty much the same thing. 

What am I doing wrong???

Thanks!