Queued Write without Response questions

[AKYR]

Hello Nordic team,

I'm using nrf52840 DK with SDK 15.3.0.

So I had an example sending data similarely to the ble_app_uart_c example that worked OK. 

Basically I have a TX_FIFO with about 1KB data in it and I'm processing the data 20 bytes a time as Write without Response every time I received a BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE event. 

I needed to speed up the process so I looked into the documentation and saw that the Write without Response that can be queued : https://infocenter.nordicsemi.com/index.jsp?topic=%2Fcom.nordic.infocenter.s140.api.v6.1.1%2Fgroup___b_l_e___g_a_t_t_c___f_u_n_c_t_i_o_n_s.html&anchor=ga90298b8dcd8bbe7bbe69d362d1133378

I followed the described procedure from the documentation:

• Store initial queue element count in a variable.
• Decrement the variable, which stores the currently available queue element count, by one when a call to this function returns NRF_SUCCESS.
• Increment the variable, which stores the current available queue element count, by the count variable in BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE event.

And tests showed that I have 4 buffers availables (I can call sd_ble_gattc_write 4 times before NRF_ERROR_RESOURCES) whereas ble_gattc.h shows : 

/** @defgroup BLE_GATTC_DEFAULTS GATT Client defaults
 * @{ */
#define BLE_GATTC_WRITE_CMD_TX_QUEUE_SIZE_DEFAULT 1 /**< Default number of Write without Response that can be queued for transmission. */

But this variable doesn't seem to affect the number of buffers in my example, so I guess it is overwritten somewhere but I can't find where. 

So here are my questions: 

How to find where  BLE_GATTC_WRITE_CMD_TX_QUEUE_SIZE_DEFAULT overwritten in my example ? 

Do you have an example using the procedure describe in the documentation ? (If not I have a working example but some questions about how to optimize it)

Thanks in advance,

Aloïs KYROU.

[Vidar Berg]

Hi Aloïs,

The ble_gattc_conn_cfg_t::write_cmd_tx_queue_size value is "The guaranteed minimum number of Write without Response 
that can be queued for transmission". To increase the default size you can try to add the following code to ble_stack_init:
static void ble_stack_init(void)
{
    ret_code_t err_code;

    err_code = nrf_sdh_enable_request();
    APP_ERROR_CHECK(err_code);

    // Configure the BLE stack using the default settings.
    // Fetch the start address of the application RAM.
    uint32_t ram_start = 0;
    err_code = nrf_sdh_ble_default_cfg_set(APP_BLE_CONN_CFG_TAG, &ram_start);
    APP_ERROR_CHECK(err_code);


    ble_cfg_t ble_cfg;

    memset(&ble_cfg, 0, sizeof(ble_cfg));
    ble_cfg.conn_cfg.conn_cfg_tag = APP_BLE_CONN_CFG_TAG;
    ble_cfg.conn_cfg.params.gattc_conn_cfg.write_cmd_tx_queue_size = 7;

    err_code = sd_ble_cfg_set(BLE_CONN_CFG_GATTC, &ble_cfg, ram_start);
    APP_ERROR_CHECK(err_code);
    
    ...


Note: you may need to increase the app RAM base to leave more RAM to the Softdevice - See debug log for details. 

Best regards,
Vidar

[AKYR]

Hello Vidar and thanks for your answer,

I have made this work by waiting for the write_cmd_tx_queue_size number of buffers to be released before looping on sd_ble_gattc_write again. 

I would like to optimize this by doing a sd_ble_gattc_write whenever a buffer is available instead of waiting for the pool to be available. 

To do so I need to protect the calls to the function processing the buffers. I tried using sd_mutex functions but i got undefined behavior. Looks like the thread managing the BLE event is forever locked... 

If you have some suggestion, here is the sample of the code that works:

#define MINIMUM_AVAILABLE_WRITE_CMD_TX_BUFFER 4
uint8_t m_available_write_cmd_tx_buffer;

static void on_write_cmd_tx_rsp(const ble_gattc_evt_t *const p_ble_gattc_evt)
{
  ret_code_t err_code;
  const ble_gattc_evt_write_cmd_tx_complete_t *p_write_cmd_tx_rsp = &(p_ble_gattc_evt->params.write_cmd_tx_complete);
  m_available_write_cmd_tx_buffer += p_write_cmd_tx_rsp->count;
  if (m_available_write_cmd_tx_buffer >= MINIMUM_AVAILABLE_WRITE_CMD_TX_BUFFER)
  {
    err_code = process_tx_fifo_send_bytes();
    APP_ERROR_CHECK(err_code);
  }
}

static uint32_t process_tx_fifo_send_bytes()
{
  uint32_t nbReadableBytes = 0;
  uint32_t length = 0;
  static uint8_t data_array[BLE_MLDP_MAX_DATA_LEN];
  ret_code_t err_code;
  bool exitMainLoop = false;

  // Get nbReadableBytes availables in the FIFO
  err_code = app_fifo_read(&m_mldp_tx_fifo, NULL, &nbReadableBytes);
  if ((err_code != NRF_ERROR_NOT_FOUND) && (err_code != NRF_SUCCESS))
  {
    APP_ERROR_CHECK(err_code);
  }

  if ((nbReadableBytes == 0) || err_code == NRF_ERROR_NOT_FOUND)
  {
    // No data to send in TX FIFO
    return NRF_SUCCESS;
  }

  while (m_available_write_cmd_tx_buffer && nbReadableBytes)
  {
    length = fmin(nbReadableBytes, BLE_MLDP_MAX_DATA_LEN);

    err_code = app_fifo_read(&m_mldp_tx_fifo, data_array, &length);
    if ((err_code != NRF_ERROR_NOT_FOUND) && (err_code != NRF_SUCCESS))
    {
      APP_ERROR_CHECK(err_code);
    }

    do
    {
      err_code = ble_mldps_c_string_send(&m_ble_mldps_c, data_array, length);
      if ((err_code != NRF_SUCCESS) && (err_code != NRF_ERROR_BUSY))
      {
        NRF_LOG_ERROR("Failed sending MLDP message. Error 0x%x. ", err_code);
        APP_ERROR_CHECK(err_code);
      }
    } while (err_code == NRF_ERROR_BUSY);

    if (err_code == NRF_SUCCESS)
    {
      m_available_write_cmd_tx_buffer--;
    }
    else if (err_code == NRF_ERROR_RESOURCES)
    {
      // Should reeval m_available_write_cmd_tx_buffer probably but will see this later
      break;
    }
    else
    {
      APP_ERROR_CHECK(err_code);
    }
    nbReadableBytes -= length;
  }
  return NRF_SUCCESS;
}

static void ble_evt_handler(ble_evt_t const *p_ble_evt, void *p_context)
{
  ret_code_t err_code;
  ble_gap_evt_t const *p_gap_evt = &p_ble_evt->evt.gap_evt;

  pm_handler_secure_on_connection(p_ble_evt);

  switch (p_ble_evt->header.evt_id)
  {
  case BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE:
  {
    on_write_cmd_tx_rsp(&(p_ble_evt->evt.gattc_evt));
    APP_ERROR_CHECK(err_code);
  }
  break;

  default:
    break;
  }
}

int main(void)
{
  //Inits the FIFO with 1K bytes data
  init_fifo(&m_mldp_tx_fifo);
  //Inits write_cmd_tx_queue_size with the value MINIMUM_AVAILABLE_WRITE_CMD_TX_BUFFER
  ble_stack_init();
  //First call, the next will be from callback from BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE event
  process_tx_fifo_send_bytes();
}

And here is what I modified to try to make work:

static void on_write_cmd_tx_rsp(const ble_gattc_evt_t *const p_ble_gattc_evt)
{
    ret_code_t err_code;
    const ble_gattc_evt_write_cmd_tx_complete_t *p_write_cmd_tx_rsp = &(p_ble_gattc_evt->params.write_cmd_tx_complete);
    // mutex inited in main function
    while (sd_mutex_acquire(&m_mutex) == NRF_ERROR_SOC_MUTEX_ALREADY_TAKEN)
    {
        nrf_delay_us(1000);
    }
    m_available_write_cmd_tx_buffer += p_write_cmd_tx_rsp->count;
    /*if (m_available_write_cmd_tx_buffer >= MINIMUM_AVAILABLE_WRITE_CMD_TX_BUFFER)
    {*/
        err_code = process_tx_fifo_send_bytes();
        APP_ERROR_CHECK(err_code);
        sd_mutex_release(&m_mutex);
    //}
}

My guess is there is something wrong with the way I'm waiting ...

Do you have an advice on how to do this correctly ? 

Thanks in advance,

Aloïs KYROU

[Vidar Berg]

Hello Aloïs,

Did you consider to just process the buffer once when you insert a new data, and then each time you get the BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE event? This should ensure sufficient utilization of the Softdevice's transmit buffer.

AKYR said:

I would like to optimize this by doing a sd_ble_gattc_write whenever a buffer is available instead of waiting for the pool to be available. 

 BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE would signalize that you have buffer space available.

Best regards,

Vidar

[AKYR]

Yes this was my first implementation that worked.

But I need it to go faster to match the speed of a previous device.

Working on the 4 buffers already upgraded the speed by ~250%.

But waiting on the 4 buffers to be released to write again 4 times can be optimize and it is more or less what I'm lacking to match the speed I need :) 

When you write do the 4 consecutive writes at the beggining you have a buffer release each 2-3 seconds whereas when working on one buffer only there is a release every 6-7 seconds.

[Vidar Berg]

There should be plenty of time to prepare the buffers for the next connection event when you get the tx complete signal. So I don't think you will be able to further improve the throughput by changing the buffer handling. An easy way to confirm this would be to continuously call the send from your main loop and ignore the return value to see how much data you can push through.

Are you testing with Nordic nRF devices on both sides? The connection event length can have a big impact on how many packets you can send per event. 

Below is a sniffer trace from a test I did with coded PHY. Notice the number of packets between the connection intervals: