SDK 17 - NRF52840 - nrf_queue_write() asserts when buffer size > queue size

[Nikosant03]

Hi everyone,

I use the function nrf_queue_write for queueing data. Also I use the NRF_QUEUE_MODE_OVERFLOW  mode to overwrite old data in case the queue is full.

My problem is that nrf_queue_write asserts when the buffer size is greater thant the queue size (see nrf_queue.c)

ret_code_t nrf_queue_write(nrf_queue_t const * p_queue,
                           void const        * p_data,
                           size_t              element_count)
{
    ret_code_t status = NRF_SUCCESS;

    ASSERT(p_queue != NULL);
    ASSERT(p_data != NULL);
    ASSERT(element_count <= p_queue->size);
    
.......
.......
.......

So when I declare a queue size of 4 and try to queue a buffer of 5 elements, nrf_queue_write asserts.

#define QUEUE_SIZE 4                                // 4 element queue

NRF_QUEUE_DEF(int16_t, m_sensor_data_queue, QUEUE_SIZE, NRF_QUEUE_MODE_OVERFLOW); 

int16_t data_in[] = {6000, 8000, -10000, -12000, -14000};
uint8_t data_in_size = sizeof(data_in) / sizeof(data_in[0]);

void my_sensors_buffer() {
  ret_code_t err_code;

  nrf_queue_write(&m_sensor_data_queue, data_in, data_in_size); // It asserts when data_in_size > QUEUE_SIZE 
  APP_ERROR_CHECK(err_code);

}

It seems that something I don't understand. How can I work this arround?

[Einar Thorsrud]

Hi,

So when I declare a queue size of 4 and try to queue a buffer of 5 elements, nrf_queue_write asserts.

I think this assert makes sense. Even if you use NRF_QUEUE_MODE_OVERFLOW attempting to queue more elements than there is room for in the buffer seems like an indication of a bug in the caller code.

How can I work this arround?

I suggest you handle this in your code so that you avoid this issue in the first place. If you really want to be able to write more elements than you have scaled the queue for and to automatically discard some elements I suggest you do that in your code before calling nrf_queue_write().

[Nikosant03]

Thank you for your answer Einar,

Einar Thorsrud said:

I think this assert makes sense. Even if you use NRF_QUEUE_MODE_OVERFLOW attempting to queue more elements than there is room for in the buffer seems like an indication of a bug in the caller code.

So what is the purpose of NRF_QUEUE_MODE_OVERFLOW? The documentation says that old data will be overwritten in case of overflow. Trying to queue more elements than the queue size doesn't lead to overflow condition?

[Einar Thorsrud]

Nikosant03 said:

So what is the purpose of NRF_QUEUE_MODE_OVERFLOW? The documentation says that old data will be overwritten in case of overflow. Trying to queue more elements than the queue size doesn't lead to overflow condition?

You are right. But this does not work when you try to queue more elements in a single write. I think this assert makes sense as this typically indicates an error.

(This option (NRF_QUEUE_MODE_OVERFLOW) will still handle overwriting old elements if for instance the queue size was 4, there were 2 elements in the queue and you try to add 3.)

[Nikosant03]

You are right!! I just tried to push two array back to back and the overflow condition works!!

However, when I pop the elements I do not get what I expect.. After pushing both arrays I expect the queue to be:

[-12000 -14000 16000 18000 20000 2000 2500 2800 3000 4000]

If you check the logs below you will notice that the first two logs are ok and then the queue changes to:

[2800 6000 8000 -10000 -12000 -14000 16000 18000 3000 4000]

That's quite strange.. I am I doing something wrong during pushing or popping?

#define QUEUE_SIZE 10                               

NRF_QUEUE_DEF(int16_t, m_sensor_data_queue, QUEUE_SIZE, NRF_QUEUE_MODE_OVERFLOW); // define a queue instance

static int16_t data_in[] = {6000, 8000, -10000, -12000, -14000, 16000, 18000, 20000};
static int16_t data_in_2[] = {2000, 2500, 2800, 3000, 4000};
int16_t data_out[10];
static uint8_t data_in_size = sizeof(data_in) / sizeof(data_in[0]);
static uint8_t data_in_size_2 = sizeof(data_in_2) / sizeof(data_in_2[0]);
uint8_t counter=0;

void my_sensors_buffer() {
  ret_code_t err_code;
 
  NRF_LOG_INFO("Queue some data");
  NRF_LOG_INFO("Byte size of array: %d", sizeof(data_in));
  NRF_LOG_INFO("Element size of array: %d", data_in_size);

  if (data_in_size > QUEUE_SIZE) {
    NRF_LOG_INFO("Warning: data input > queue size")
  }

   nrf_queue_write(&m_sensor_data_queue, data_in, data_in_size);
   APP_ERROR_CHECK(err_code);

   nrf_queue_write(&m_sensor_data_queue, data_in_2, data_in_size_2);
   APP_ERROR_CHECK(err_code);

  for (uint16_t i = 0; i < QUEUE_SIZE; i++) {
    err_code = nrf_queue_pop(&m_sensor_data_queue, data_out + i); // Pop element from the front of the queue. By default the return element is removed from the queue
    APP_ERROR_CHECK(err_code);
  }

  for (uint16_t i = 0; i < QUEUE_SIZE; i++) {
    NRF_LOG_INFO("Counter %d - Pop[%d] %d",counter, i, *(data_out + i));
  }
  counter++;
}

<info> app: Queue buffer example started..
<info> app: Queue some data
<info> app: Byte size of array: 16
<info> app: Element size of array: 8
<info> app: Counter 0 - Pop[0] -12000
<info> app: Counter 0 - Pop[1] -14000
<info> app: Counter 0 - Pop[2] 16000
<info> app: Counter 0 - Pop[3] 18000
<info> app: Counter 0 - Pop[4] 20000
<info> app: Counter 0 - Pop[5] 2000
<info> app: Counter 0 - Pop[6] 2500
<info> app: Counter 0 - Pop[7] 2800
<info> app: Counter 0 - Pop[8] 3000
<info> app: Counter 0 - Pop[9] 4000
<info> app: Queue some data
<info> app: Byte size of array: 16
<info> app: Element size of array: 8
<info> app: Counter 1 - Pop[0] -12000
<info> app: Counter 1 - Pop[1] -14000
<info> app: Counter 1 - Pop[2] 16000
<info> app: Counter 1 - Pop[3] 18000
<info> app: Counter 1 - Pop[4] 20000
<info> app: Counter 1 - Pop[5] 2000
<info> app: Counter 1 - Pop[6] 2500
<info> app: Counter 1 - Pop[7] 2800
<info> app: Counter 1 - Pop[8] 3000
<info> app: Counter 1 - Pop[9] 4000
<info> app: Queue some data
<info> app: Byte size of array: 16
<info> app: Element size of array: 8
<info> app: Counter 2 - Pop[0] 2800
<info> app: Counter 2 - Pop[1] 6000
<info> app: Counter 2 - Pop[2] 8000
<info> app: Counter 2 - Pop[3] -10000
<info> app: Counter 2 - Pop[4] -12000
<info> app: Counter 2 - Pop[5] -14000
<info> app: Counter 2 - Pop[6] 16000
<info> app: Counter 2 - Pop[7] 18000
<info> app: Counter 2 - Pop[8] 3000
<info> app: Counter 2 - Pop[9] 4000
<info> app: Queue some data
<info> app: Byte size of array: 16
<info> app: Element size of array: 8
<info> app: Counter 3 - Pop[0] -12000
<info> app: Counter 3 - Pop[1] -14000
<info> app: Counter 3 - Pop[2] 16000
<info> app: Counter 3 - Pop[3] 18000
<info> app: Counter 3 - Pop[4] 20000
<info> app: Counter 3 - Pop[5] 2000
<info> app: Counter 3 - Pop[6] 2500
<info> app: Counter 3 - Pop[7] 2800
<info> app: Counter 3 - Pop[8] 3000
<info> app: Counter 3 - Pop[9] 4000
<info> app: Queue some data
<info> app: Byte size of array: 16
<info> app: Element size of array: 8
<info> app: Counter 4 - Pop[0] 2800
<info> app: Counter 4 - Pop[1] 6000
<info> app: Counter 4 - Pop[2] 8000
<info> app: Counter 4 - Pop[3] -10000
<info> app: Counter 4 - Pop[4] -12000
<info> app: Counter 4 - Pop[5] -14000
<info> app: Counter 4 - Pop[6] 16000
<info> app: Counter 4 - Pop[7] 18000
<info> app: Counter 4 - Pop[8] 3000
<info> app: Counter 4 - Pop[9] 4000
<info> app: Queue some data
<info> app: Byte size of array: 16
<info> app: Element size of array: 8
<info> app: Counter 5 - Pop[0] -12000
<info> app: Counter 5 - Pop[1] -14000
<info> app: Counter 5 - Pop[2] 16000
<info> app: Counter 5 - Pop[3] 18000
<info> app: Counter 5 - Pop[4] 20000
<info> app: Counter 5 - Pop[5] 2000
<info> app: Counter 5 - Pop[6] 2500
<info> app: Counter 5 - Pop[7] 2800
<info> app: Counter 5 - Pop[8] 3000
<info> app: Counter 5 - Pop[9] 4000
<info> app: Queue some data
<info> app: Byte size of array: 16
<info> app: Element size of array: 8
<info> app: Counter 6 - Pop[0] 2800
<info> app: Counter 6 - Pop[1] 6000
<info> app: Counter 6 - Pop[2] 8000
<info> app: Counter 6 - Pop[3] -10000
<info> app: Counter 6 - Pop[4] -12000
<info> app: Counter 6 - Pop[5] -14000
<info> app: Counter 6 - Pop[6] 16000
<info> app: Counter 6 - Pop[7] 18000
<info> app: Counter 6 - Pop[8] 3000
<info> app: Counter 6 - Pop[9] 4000
<info> app: Queue some data
<info> app: Byte size of array: 16
<info> app: Element size of array: 8
<info> app: Counter 7 - Pop[0] -12000
<info> app: Counter 7 - Pop[1] -14000
<info> app: Counter 7 - Pop[2] 16000
<info> app: Counter 7 - Pop[3] 18000
<info> app: Counter 7 - Pop[4] 20000
<info> app: Counter 7 - Pop[5] 2000
<info> app: Counter 7 - Pop[6] 2500
<info> app: Counter 7 - Pop[7] 2800
<info> app: Counter 7 - Pop[8] 3000
<info> app: Counter 7 - Pop[9] 4000

#define QUEUE_SIZE 10                               

NRF_QUEUE_DEF(int16_t, m_sensor_data_queue, QUEUE_SIZE, NRF_QUEUE_MODE_OVERFLOW); // define a queue instance

static int16_t data_in[] = {6000, 8000, -10000, -12000, -14000, 16000, 18000, 20000};
static int16_t data_in_2[] = {2000, 2500, 2800, 3000, 4000};
int16_t data_out[10];
static uint8_t data_in_size = sizeof(data_in) / sizeof(data_in[0]);
static uint8_t data_in_size_2 = sizeof(data_in_2) / sizeof(data_in_2[0]);
uint8_t counter=0;

void my_sensors_buffer() {
  ret_code_t err_code;
 
  NRF_LOG_INFO("Queue some data");
  NRF_LOG_INFO("Byte size of array: %d", sizeof(data_in));
  NRF_LOG_INFO("Element size of array: %d", data_in_size);

  if (data_in_size > QUEUE_SIZE) {
    NRF_LOG_INFO("Warning: data input > queue size")
  }

   nrf_queue_write(&m_sensor_data_queue, data_in, data_in_size);
   APP_ERROR_CHECK(err_code);

   nrf_queue_write(&m_sensor_data_queue, data_in_2, data_in_size_2);
   APP_ERROR_CHECK(err_code);

  for (uint16_t i = 0; i < QUEUE_SIZE; i++) {
    err_code = nrf_queue_pop(&m_sensor_data_queue, data_out + i); // Pop element from the front of the queue. By default the return element is removed from the queue
    APP_ERROR_CHECK(err_code);
  }

  for (uint16_t i = 0; i < QUEUE_SIZE; i++) {
    NRF_LOG_INFO("Counter %d - Pop[%d] %d",counter, i, *(data_out + i));
  }
  counter++;
}

[Einar Thorsrud]

Hi,

There are a few issues with your test code, like wrong types and checking error values that were never written to. I took the liberty to clean it up a bit while maintaining the intended functionality:

#define QUEUE_SIZE 10
NRF_QUEUE_DEF(int16_t, m_sensor_data_queue, QUEUE_SIZE, NRF_QUEUE_MODE_OVERFLOW);

static int16_t data_in[] = {10, 20, 30, 40, 50, 60, 70, 80};
static int16_t data_in_2[] = {1, 2, 3, 4, 5};

void my_sensors_buffer(int run_count)
{
    ret_code_t err_code;
    int16_t data_out;

    for(int run = 0; run < run_count; run++)
    {
        NRF_LOG_INFO("Queue some data, run %i", run);

        err_code = nrf_queue_write(&m_sensor_data_queue, data_in, ARRAY_SIZE(data_in));
        APP_ERROR_CHECK(err_code);

        err_code = nrf_queue_write(&m_sensor_data_queue, data_in_2, ARRAY_SIZE(data_in_2));
        APP_ERROR_CHECK(err_code);

        for (uint16_t i = 0; i < QUEUE_SIZE; i++)
        {
            err_code = nrf_queue_pop(&m_sensor_data_queue, &data_out);
            APP_ERROR_CHECK(err_code);
            NRF_LOG_INFO("Pop[%d] %d", run, data_out);
        }

        nrf_delay_ms(100); // Delay a bit to have time to output logs..
    }
}

Testing that I see the same as you, and this does not seem like the expected behavior. I need to look more into it to understand what is going on here, though.

Edit: simplified a bit and use other numbers, add resulting output after 9 iterations:

<info> app: Queue some data, run 0
<info> app: Pop[0] 40
<info> app: Pop[0] 50
<info> app: Pop[0] 60
<info> app: Pop[0] 70
<info> app: Pop[0] 80
<info> app: Pop[0] 1
<info> app: Pop[0] 2
<info> app: Pop[0] 3
<info> app: Pop[0] 4
<info> app: Pop[0] 5
<info> app: Queue some data, run 1
<info> app: Pop[1] 40
<info> app: Pop[1] 50
<info> app: Pop[1] 60
<info> app: Pop[1] 70
<info> app: Pop[1] 80
<info> app: Pop[1] 1
<info> app: Pop[1] 2
<info> app: Pop[1] 3
<info> app: Pop[1] 4
<info> app: Pop[1] 5
<info> app: Queue some data, run 2
<info> app: Pop[2] 3
<info> app: Pop[2] 10
<info> app: Pop[2] 20
<info> app: Pop[2] 30
<info> app: Pop[2] 40
<info> app: Pop[2] 50
<info> app: Pop[2] 60
<info> app: Pop[2] 70
<info> app: Pop[2] 4
<info> app: Pop[2] 5
<info> app: Queue some data, run 3
<info> app: Pop[3] 40
<info> app: Pop[3] 50
<info> app: Pop[3] 60
<info> app: Pop[3] 70
<info> app: Pop[3] 80
<info> app: Pop[3] 1
<info> app: Pop[3] 2
<info> app: Pop[3] 3
<info> app: Pop[3] 4
<info> app: Pop[3] 5
<info> app: Queue some data, run 4
<info> app: Pop[4] 3
<info> app: Pop[4] 10
<info> app: Pop[4] 20
<info> app: Pop[4] 30
<info> app: Pop[4] 40
<info> app: Pop[4] 50
<info> app: Pop[4] 60
<info> app: Pop[4] 70
<info> app: Pop[4] 4
<info> app: Pop[4] 5
<info> app: Queue some data, run 5
<info> app: Pop[5] 40
<info> app: Pop[5] 50
<info> app: Pop[5] 60
<info> app: Pop[5] 70
<info> app: Pop[5] 80
<info> app: Pop[5] 1
<info> app: Pop[5] 2
<info> app: Pop[5] 3
<info> app: Pop[5] 4
<info> app: Pop[5] 5
<info> app: Queue some data, run 6
<info> app: Pop[6] 3
<info> app: Pop[6] 10
<info> app: Pop[6] 20
<info> app: Pop[6] 30
<info> app: Pop[6] 40
<info> app: Pop[6] 50
<info> app: Pop[6] 60
<info> app: Pop[6] 70
<info> app: Pop[6] 4
<info> app: Pop[6] 5
<info> app: Queue some data, run 7
<info> app: Pop[7] 40
<info> app: Pop[7] 50
<info> app: Pop[7] 60
<info> app: Pop[7] 70
<info> app: Pop[7] 80
<info> app: Pop[7] 1
<info> app: Pop[7] 2
<info> app: Pop[7] 3
<info> app: Pop[7] 4
<info> app: Pop[7] 5
<info> app: Queue some data, run 8
<info> app: Pop[8] 3
<info> app: Pop[8] 10
<info> app: Pop[8] 20
<info> app: Pop[8] 30
<info> app: Pop[8] 40
<info> app: Pop[8] 50
<info> app: Pop[8] 60
<info> app: Pop[8] 70
<info> app: Pop[8] 4
<info> app: Pop[8] 5

[Nikosant03]

Thank you for your effort Einar!!

[Einar Thorsrud]

There are clearly some issues with the queue implementation when using nrf_queue_write() combined with NRF_QUEUE_MODE_OVERFLOW. Another issue (which may have the same root cause) can be seen by using a different length, where the total number of elements added by the two calls to nrf_queue_write() is one more than the queue length.

See this code:

#define QUEUE_SIZE 6
NRF_QUEUE_DEF(uint32_t, m_sensor_data_queue, QUEUE_SIZE, NRF_QUEUE_MODE_OVERFLOW);

static int16_t data_in[] = {10, 20, 30, 40};
static int16_t data_in_2[] = {1, 2, 3};

void my_sensors_buffer(int run_count)
{
    ret_code_t err_code;
    int16_t data_out;
    size_t queued;

    for(int run = 0; run < run_count; run++)
    {
        NRF_LOG_INFO("Queue some data, run %i", run);

        queued = nrf_queue_utilization_get(&m_sensor_data_queue);
        NRF_LOG_INFO("Queued before first write: %i", queued);

        err_code = nrf_queue_write(&m_sensor_data_queue, data_in, ARRAY_SIZE(data_in));
        APP_ERROR_CHECK(err_code);

        queued = nrf_queue_utilization_get(&m_sensor_data_queue);
        NRF_LOG_INFO("Queued before second write: %i", queued);

        err_code = nrf_queue_write(&m_sensor_data_queue, data_in_2, ARRAY_SIZE(data_in_2));
        APP_ERROR_CHECK(err_code);

        queued = nrf_queue_utilization_get(&m_sensor_data_queue);
        NRF_LOG_INFO("Queued after both writes: %i", queued);

        for (uint16_t i = 0; i < QUEUE_SIZE; i++)
        {
            err_code = nrf_queue_pop(&m_sensor_data_queue, &data_out);
            APP_ERROR_CHECK(err_code);
            NRF_LOG_INFO("Pop[%d] %d", run, data_out);
        }

        nrf_delay_ms(100); // Delay a bit to have time to output logs..
    }
}

Which gives this output:

<info> app: Queue some data, run 0
<info> app: Queued before first write: 0
<info> app: Queued before second write: 4
<info> app: Queued after both writes: 0
<error> app: ERROR 5 [NRF_ERROR_NOT_FOUND] at C:\Users\eith\SDK\nRF5_SDK_17.1.0_ddde560\examples\other_projects\nrf_queue_test_279738\main.c:46
PC at: 0x0002D02D
<error> app: End of error report

Note that the queue utilization after the first write is 4, as expected, but after the second write it is 0, and not 6 which it should have been (where one element should have been discarded).

I have not got to the bottom of this, but for now I suggest using nrf_queue_push() instead. Then you could for instance replace:

        err_code = nrf_queue_write(&m_sensor_data_queue, data_in, ARRAY_SIZE(data_in));
        APP_ERROR_CHECK(err_code);

with:

        for (uint16_t i = 0; i < ARRAY_SIZE(data_in); i++)
        {
            err_code = nrf_queue_push(&m_sensor_data_queue, &data_in[i]);
            APP_ERROR_CHECK(err_code);
        }

This is an effective workaround of all issues I have seen.

[Nikosant03]

Hi Einar,

Thank you for your effort, I've just confirmed that nrf_queue_push() works flawlessly combined with NRF_QUEUE_MODE_OVERFLOW mode. I will go ahead with this approach.

Do you know if these bugs are fixed by the development team in NRF connect SDK?

Nick

[Einar Thorsrud]

Hi Nick,

The nRF Connect SDK has a completely different architecture and use Zephyr queues (FIFO would be equivalent ish).