Increasing BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT

[CktDesigner]

I'm attempting to increase the size of the TX Queue in an effort to increase notification performance.   

I'm using nRF52840 with an S140 soft device.   I use Segger Embedded Studio and am using the Laird Connectivity BL-654 DVK evaluation board.

The following defines have been modified in sdk_config.h

#define NRF_SDH_BLE_GAP_DATA_LENGTH 251

#define NRF_SDH_BLE_GAP_EVENT_LENGTH 320

#define NRF_SDH_BLE_GATT_MAX_MTU_SIZE 247

I call:

void connEvtLenExtSet(bool onOff) {
  ret_code_t err_code;
  ble_opt_t opt;

  memset(&opt,0,sizeof(opt));
  opt.common_opt.conn_evt_ext.enable= onOff ? 1 : 0;

  err_code= sd_ble_opt_set(BLE_COMMON_OPT_CONN_EVT_EXT, &opt);
  APP_ERROR_CHECK(err_code);
}

after gap and gatts are initialized:

    // Configure and initialize the BLE stack.
    ble_stack_init();

    // Initialize modules.
    timers_init();
    buttons_leds_init(&erase_bonds);
    gap_params_init();
    gatt_init();
    //dataLenExtSet();
    connEvtLenExtSet(true);
    advertising_init();
    services_init();
    sensor_simulator_init();
    conn_params_init();
    peer_manager_init();

    // Initialize the Queues
    bleNotificationsWritten= 0;
    bleNotificationsSent= 0;
    bleNotificationsFailed= 0;
    bleNotificationsReceived= 0;
    bleNotificationsAvailable= BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT;

This code also initializes some counters to keep track of notifications, including "bleNotificationsAvailable" that should monitor how many TX Queue entries are available.

I update the Phy to use 2MBPS using:

void updatePhy(uint16_t handle) {
  ret_code_t err_code;
  ble_gap_phys_t phys = {
    //.tx_phys = BLE_GAP_PHY_2MBPS | BLE_GAP_PHY_1MBPS,
    //.rx_phys = BLE_GAP_PHY_2MBPS | BLE_GAP_PHY_1MBPS

    .tx_phys = BLE_GAP_PHY_2MBPS,
    .rx_phys = BLE_GAP_PHY_2MBPS
  };
  NRF_LOG_DEBUG("PHY update set.");

  err_code= sd_ble_gap_phy_update(handle, &phys);
  APP_ERROR_CHECK(err_code);
}

This is called after the connection is established:

        case BLE_GAP_EVT_CONNECTED:
            NRF_LOG_INFO("Connected");
            err_code = bsp_indication_set(BSP_INDICATE_CONNECTED);
            APP_ERROR_CHECK(err_code);
            m_conn_handle = p_ble_evt->evt.gap_evt.conn_handle;
            err_code = nrf_ble_qwr_conn_handle_assign(&m_qwr, m_conn_handle);
            APP_ERROR_CHECK(err_code);
            updatePhy(m_conn_handle);
            break;

I keep track of the number of notifications being processed using the variable "bleNotificationsAvailable".   It is initialized to BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT.

I changed BLE_GATTS_HVN_TX_QUEUE_SIZE_DEFAULT to 20 in ble_gatts.h

This variable is incremented by the number of notifications sent in the BLE callback:

        case BLE_GATTS_EVT_HVN_TX_COMPLETE:
            bleNotificationsSent= bleNotificationsSent + p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count;
            bleNotificationsAvailable= bleNotificationsAvailable + p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count;
            NRF_LOG_DEBUG("HVN_TX_COMPLETE: Packets: %d",p_ble_evt->evt.gatts_evt.params.hvn_tx_complete.count);
            // Wake the Send process
            xTaskNotify(SendTaskHandle,SEND_PACKET_QUEUED,eSetBits);
            break;

        case BLE_GATTS_EVT_HVC:
            bleNotificationsReceived++;
            break;

and is decremented when a notification is requested to be sent:

Since Notifications can be sent by multiple sensors, the "Send" code is consolidated in one FreeRTOS task.   This is the body of the task:

for (;;) {
    // Block until something appears in the queue...
    // Only bit that should cause Notification is SEND_PACKET_QUEUED, so don't even check...
    xTaskNotifyWait(0x00,             // Don't clear any bits on entry
                    ~0x00,            // Clear all bits on exit
                    &bleSendNotifyBits,
                    portMAX_DELAY);   // Block indefinitely
    
    // While there is space in the BLE queue AND there are packets to send...
    while(    (bleNotificationsAvailable > 2) 
           && (uxQueueMessagesWaiting(characteristicQueue) > 0) ) {
      // Still room in TX QUEUE and a characteristic is waiting in the queue
      // Read the queue item
      xStatus= xQueueReceive(characteristicQueue,&charToSend,0);
      if( xStatus == pdPASS ) {	// Read from queue was successful
        ble_gatts_hvx_params_t hvx_params;

        memset(&hvx_params, 0, sizeof(ble_gatts_hvx_params_t));

        size                   = charToSend.size;

        hvx_params.handle      = charToSend.charHandle;
        hvx_params.type        = BLE_GATT_HVX_NOTIFICATION;
        hvx_params.offset      = 0;
        hvx_params.p_len       = &size;
        hvx_params.p_data      = charToSend.charPtr;

        err_code= sd_ble_gatts_hvx(charToSend.connHandle,&hvx_params);
        // size will reflect the number of bytes sent
        if(    (err_code == NRF_SUCCESS)
            && (charToSend.size == size)
          ) {
          bleNotificationsAvailable--;
        } else {
          bleNotificationsFailed++;
          NRF_LOG_DEBUG("Packet Send Failure: %d : %d",err_code,bleNotificationsAvailable);
          break;
          //APP_ERROR_CHECK(err_code);
        }
      } else {
        break;
      }
    }
  }

This task will send Notifications as long as there is TX queue space available and as long as a notification is present in the queue (from other tasks that handle sensors).

I thought using this code would eliminate getting NRF_ERROR_RESOURCES errors, but I still get them.  If I increase the connect intervals (from 7.5 ms to 15 or 30 ms) the number of errors increases.

When an error occurs, the error code indicates NRF_ERROR_RESOURCES and the value of bleNotificationsAvailable is 16.    So (theoretically) there should be queue space.

Here is the code where the intervals are defined:

#define MIN_CONN_INTERVAL                   MSEC_TO_UNITS(7.5, UNIT_1_25_MS)        /**< Minimum acceptable connection interval (0.4 seconds). */
#define MAX_CONN_INTERVAL                   MSEC_TO_UNITS(650, UNIT_1_25_MS)        /**< Maximum acceptable connection interval (0.65 second). */

I though that perhaps there was some sort of race condition, so I increased the comparison (as shown above) for the number of free TX queue entries to 2.   So when I activate the "Send" task, there should be at least 2 free TX queue slots or no call to sd_ble_gatts_hvn should be made.

However, I still get a lot of debug print statements indicating packet failures with NRF_ERROR_RESOURCES and supposedly 16 empty TX queue slots.

For completion, here is the code for a testing task that sends packets.   For now I'm just using one that sends a packet about every 7.6 ms.   The characteristic size is 244 bytes.

    // Send value if connected and notifying
    if (service->conn_handle != BLE_CONN_HANDLE_INVALID) {

        charQueueItem          thisChar;

        thisChar.connHandle      = service->conn_handle;
        thisChar.charHandle      = service->mic_data_handles.value_handle;
        thisChar.size            = sizeof(micDataCharacteristic);
        thisChar.charPtr         = (uint8_t *)charToSend;

        // Send the packet info
        xStatus= xQueueSendToBack(characteristicQueue, &thisChar, 0);
        if(xStatus != pdPASS ) {
          itemsInQueue= uxQueueMessagesWaiting(characteristicQueue);
          //fprintfsock("Could not send MIC packet to the queue.\n");
          while(1);
        }
        // Increment the written count
        bleNotificationsWritten++;
        // Wake the Send process
        xTaskNotify(SendTaskHandle,SEND_PACKET_QUEUED,eSetBits);

    } else {
        err_code = NRF_ERROR_INVALID_STATE;
    }

This creates a FreeRTOS queue item that includes information about the connection handle, the characteristic handle, a pointer to the characteristic value to be sent, etc. and sends it to the queue.   It also wakes the Send task to actually send the notification if conditions permit.     If the notification can't be sent, the queue item won't be consumed.   When a BLE_GATTS_EVT_HVN_TX_COMPLETE arrives, the bleNotificationsAvailable variable is updated and the Send Task is notified to check if anything should be sent.

I didn't expect to see the NRF_ERROR_RESOURCES.   Are there any other conditions (besides the TX QUEUE being overrun) that I should consider?   My variable indicates that there are 16 slots available when the error occurs.

I am attempting to receive the notifications on a Samsung S8 running Android 9.   I have used LightBlue, nRF Connect, and my own Android app as receiving apps (central) and all do pretty much the same thing. 

I'm not getting anywhere near the data rate I expected.   I need to send 244 bytes every 7.5 ms, which is just over 256 Kbps.   Shouldn't this be possible with BLE 5.x?

What am I doing wrong???

Thanks!

[Kenneth]

To narrow down the cause (phone or peripheral) I think we need an on-air sniffer trace. It should be possible to use https://www.nordicsemi.com/Software-and-tools/Development-Tools/nRF-Sniffer-for-Bluetooth-LE

However you will need to somehow go down on the speed (e.g. send data more infrequent and/or reduce on-air datarate), because the nRF sniffer tool can't send the data fast enough over UART->USB when both DLE and 2M PHY is used.  So you will loose a lot of packets, but it may be good enough to see if it's indeed phone or peripheral that are messing it up.

Best regards,
Kenneth