Queued Write without Response questions

[AKYR]

Hello Nordic team,

I'm using nrf52840 DK with SDK 15.3.0.

So I had an example sending data similarely to the ble_app_uart_c example that worked OK. 

Basically I have a TX_FIFO with about 1KB data in it and I'm processing the data 20 bytes a time as Write without Response every time I received a BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE event. 

I needed to speed up the process so I looked into the documentation and saw that the Write without Response that can be queued : https://infocenter.nordicsemi.com/index.jsp?topic=%2Fcom.nordic.infocenter.s140.api.v6.1.1%2Fgroup___b_l_e___g_a_t_t_c___f_u_n_c_t_i_o_n_s.html&anchor=ga90298b8dcd8bbe7bbe69d362d1133378

I followed the described procedure from the documentation:

• Store initial queue element count in a variable.
• Decrement the variable, which stores the currently available queue element count, by one when a call to this function returns NRF_SUCCESS.
• Increment the variable, which stores the current available queue element count, by the count variable in BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE event.

And tests showed that I have 4 buffers availables (I can call sd_ble_gattc_write 4 times before NRF_ERROR_RESOURCES) whereas ble_gattc.h shows : 

/** @defgroup BLE_GATTC_DEFAULTS GATT Client defaults
 * @{ */
#define BLE_GATTC_WRITE_CMD_TX_QUEUE_SIZE_DEFAULT 1 /**< Default number of Write without Response that can be queued for transmission. */

But this variable doesn't seem to affect the number of buffers in my example, so I guess it is overwritten somewhere but I can't find where. 

So here are my questions: 

How to find where  BLE_GATTC_WRITE_CMD_TX_QUEUE_SIZE_DEFAULT overwritten in my example ? 

Do you have an example using the procedure describe in the documentation ? (If not I have a working example but some questions about how to optimize it)

Thanks in advance,

Aloïs KYROU.

[Vidar Berg]

Hi Aloïs,

The ble_gattc_conn_cfg_t::write_cmd_tx_queue_size value is "The guaranteed minimum number of Write without Response 
that can be queued for transmission". To increase the default size you can try to add the following code to ble_stack_init:
static void ble_stack_init(void)
{
    ret_code_t err_code;

    err_code = nrf_sdh_enable_request();
    APP_ERROR_CHECK(err_code);

    // Configure the BLE stack using the default settings.
    // Fetch the start address of the application RAM.
    uint32_t ram_start = 0;
    err_code = nrf_sdh_ble_default_cfg_set(APP_BLE_CONN_CFG_TAG, &ram_start);
    APP_ERROR_CHECK(err_code);


    ble_cfg_t ble_cfg;

    memset(&ble_cfg, 0, sizeof(ble_cfg));
    ble_cfg.conn_cfg.conn_cfg_tag = APP_BLE_CONN_CFG_TAG;
    ble_cfg.conn_cfg.params.gattc_conn_cfg.write_cmd_tx_queue_size = 7;

    err_code = sd_ble_cfg_set(BLE_CONN_CFG_GATTC, &ble_cfg, ram_start);
    APP_ERROR_CHECK(err_code);
    
    ...


Note: you may need to increase the app RAM base to leave more RAM to the Softdevice - See debug log for details. 

Best regards,
Vidar

[AKYR]

Hello Vidar and thanks for your answer,

I have made this work by waiting for the write_cmd_tx_queue_size number of buffers to be released before looping on sd_ble_gattc_write again. 

I would like to optimize this by doing a sd_ble_gattc_write whenever a buffer is available instead of waiting for the pool to be available. 

To do so I need to protect the calls to the function processing the buffers. I tried using sd_mutex functions but i got undefined behavior. Looks like the thread managing the BLE event is forever locked... 

If you have some suggestion, here is the sample of the code that works:

#define MINIMUM_AVAILABLE_WRITE_CMD_TX_BUFFER 4
uint8_t m_available_write_cmd_tx_buffer;

static void on_write_cmd_tx_rsp(const ble_gattc_evt_t *const p_ble_gattc_evt)
{
  ret_code_t err_code;
  const ble_gattc_evt_write_cmd_tx_complete_t *p_write_cmd_tx_rsp = &(p_ble_gattc_evt->params.write_cmd_tx_complete);
  m_available_write_cmd_tx_buffer += p_write_cmd_tx_rsp->count;
  if (m_available_write_cmd_tx_buffer >= MINIMUM_AVAILABLE_WRITE_CMD_TX_BUFFER)
  {
    err_code = process_tx_fifo_send_bytes();
    APP_ERROR_CHECK(err_code);
  }
}

static uint32_t process_tx_fifo_send_bytes()
{
  uint32_t nbReadableBytes = 0;
  uint32_t length = 0;
  static uint8_t data_array[BLE_MLDP_MAX_DATA_LEN];
  ret_code_t err_code;
  bool exitMainLoop = false;

  // Get nbReadableBytes availables in the FIFO
  err_code = app_fifo_read(&m_mldp_tx_fifo, NULL, &nbReadableBytes);
  if ((err_code != NRF_ERROR_NOT_FOUND) && (err_code != NRF_SUCCESS))
  {
    APP_ERROR_CHECK(err_code);
  }

  if ((nbReadableBytes == 0) || err_code == NRF_ERROR_NOT_FOUND)
  {
    // No data to send in TX FIFO
    return NRF_SUCCESS;
  }

  while (m_available_write_cmd_tx_buffer && nbReadableBytes)
  {
    length = fmin(nbReadableBytes, BLE_MLDP_MAX_DATA_LEN);

    err_code = app_fifo_read(&m_mldp_tx_fifo, data_array, &length);
    if ((err_code != NRF_ERROR_NOT_FOUND) && (err_code != NRF_SUCCESS))
    {
      APP_ERROR_CHECK(err_code);
    }

    do
    {
      err_code = ble_mldps_c_string_send(&m_ble_mldps_c, data_array, length);
      if ((err_code != NRF_SUCCESS) && (err_code != NRF_ERROR_BUSY))
      {
        NRF_LOG_ERROR("Failed sending MLDP message. Error 0x%x. ", err_code);
        APP_ERROR_CHECK(err_code);
      }
    } while (err_code == NRF_ERROR_BUSY);

    if (err_code == NRF_SUCCESS)
    {
      m_available_write_cmd_tx_buffer--;
    }
    else if (err_code == NRF_ERROR_RESOURCES)
    {
      // Should reeval m_available_write_cmd_tx_buffer probably but will see this later
      break;
    }
    else
    {
      APP_ERROR_CHECK(err_code);
    }
    nbReadableBytes -= length;
  }
  return NRF_SUCCESS;
}

static void ble_evt_handler(ble_evt_t const *p_ble_evt, void *p_context)
{
  ret_code_t err_code;
  ble_gap_evt_t const *p_gap_evt = &p_ble_evt->evt.gap_evt;

  pm_handler_secure_on_connection(p_ble_evt);

  switch (p_ble_evt->header.evt_id)
  {
  case BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE:
  {
    on_write_cmd_tx_rsp(&(p_ble_evt->evt.gattc_evt));
    APP_ERROR_CHECK(err_code);
  }
  break;

  default:
    break;
  }
}

int main(void)
{
  //Inits the FIFO with 1K bytes data
  init_fifo(&m_mldp_tx_fifo);
  //Inits write_cmd_tx_queue_size with the value MINIMUM_AVAILABLE_WRITE_CMD_TX_BUFFER
  ble_stack_init();
  //First call, the next will be from callback from BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE event
  process_tx_fifo_send_bytes();
}

And here is what I modified to try to make work:

static void on_write_cmd_tx_rsp(const ble_gattc_evt_t *const p_ble_gattc_evt)
{
    ret_code_t err_code;
    const ble_gattc_evt_write_cmd_tx_complete_t *p_write_cmd_tx_rsp = &(p_ble_gattc_evt->params.write_cmd_tx_complete);
    // mutex inited in main function
    while (sd_mutex_acquire(&m_mutex) == NRF_ERROR_SOC_MUTEX_ALREADY_TAKEN)
    {
        nrf_delay_us(1000);
    }
    m_available_write_cmd_tx_buffer += p_write_cmd_tx_rsp->count;
    /*if (m_available_write_cmd_tx_buffer >= MINIMUM_AVAILABLE_WRITE_CMD_TX_BUFFER)
    {*/
        err_code = process_tx_fifo_send_bytes();
        APP_ERROR_CHECK(err_code);
        sd_mutex_release(&m_mutex);
    //}
}

My guess is there is something wrong with the way I'm waiting ...

Do you have an advice on how to do this correctly ? 

Thanks in advance,

Aloïs KYROU

[Vidar Berg]

There should be plenty of time to prepare the buffers for the next connection event when you get the tx complete signal. So I don't think you will be able to further improve the throughput by changing the buffer handling. An easy way to confirm this would be to continuously call the send from your main loop and ignore the return value to see how much data you can push through.

Are you testing with Nordic nRF devices on both sides? The connection event length can have a big impact on how many packets you can send per event. 

Below is a sniffer trace from a test I did with coded PHY. Notice the number of packets between the connection intervals:

[AKYR]

Thanks Vidar your remarks made me realise something :) 

In fact the problem wasn't in  an overlap on the process FIFO function but that it was too fast causing the other side device to lose packets ...

I'm indeed using another BLE device as peer. 

I tried sending continuously send from your main loop and ignore the return value. This way it took me 43 msec to send 1KB of data. 

When I wait for at least 4 buffers to be released, it takes 86 msecs to send 1KB of data.

The clean implementation is to call the send function when a BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE arrive to tell that a buffer is available. What I'm afraid of is that two events like this may overlap. 

Are successive BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE and other BLE events stacking in some sort of FIFO to be dispatch one after the other (Which means i can synchronously execute my function to avoid overlaps). Or does the SoftDevice implementation could send an event while the execution of another one isn't terminated yet ? 

What I mean is, is this case possible ? 

[Vidar Berg]

Glad to hear it was of some help:)

AKYR said:

In fact the problem wasn't in  an overlap on the process FIFO function but that it was too fast causing the other side device to lose packets ...

I'm not that familiar with your implementation, but just want to point out that packets that are successfully added to the output buffer (hvx function returns NRF_SUCCESS), will be re-transmitted on-air until they're acked by the peer device. And the link will be considered lost if a packet is not acked withing the supervision timeout. My point is that you may want to find what causes the packet loss on the receiving device. 

AKYR said:

The clean implementation is to call the send function when a BLE_GATTC_EVT_WRITE_CMD_TX_COMPLETE arrive to tell that a buffer is available. What I'm afraid of is that two events like this may overlap. 

They should not overlap as long as you are not using a preemptive scheduler for SD event processing. The ble callback is called in an interrupt context in most of our examples. 

[AKYR]

I can't have access to the receiving device implementation so only supositions here :) But it is using an Uart protocol tu proceed Write without Response datas. 

But ins't acking the transmition of data the point of Write with Response ? Since I'm using Write without Response I thought the packets weren't acknowledged ? 

[Vidar Berg]

All packets are acked at the link layer, but write with response will also need to be acknowledged by the application layer running on top.