Libuarte async high speed data receive and NUS transfer

[Aleksa]

Hi, 

I successfully added libuarte async library to the ble_app_uart example. Now I am able to receive data over UART from another board and my goal is to transfer it to the android application using NUS service just like in the ble_app_uart example. So here is the problem.

When you initialize the library you pass a function that will handle library events. The event I am interested in is NRF_LIBUARTE_ASYNC_EVT_RX_DATA. I get that event and it contains p_evt->data.rxtx.p_data and p_evt->data.rxtx.length that I want to send using ble_nus_data_send. Now my question when do I call the function. I tried the following things.

First I tried the most basic approach and that is to call ble_nus_data_send function right after the event is received, send the data and free the buffer.

 

        case NRF_LIBUARTE_ASYNC_EVT_RX_DATA:
        {
            do
            {
                err_code = ble_nus_data_send(&m_nus, p_evt->data.rxtx.p_data, &p_evt->data.rxtx.p_data, m_conn_handle);
                
                if ((err_code != NRF_ERROR_INVALID_STATE) &&
                    (err_code != NRF_ERROR_RESOURCES) &&
                    (err_code != NRF_ERROR_NOT_FOUND))
                {
                    APP_ERROR_CHECK(err_code);
                }
             } while (err_code == NRF_ERROR_RESOURCES);

            nrf_libuarte_async_rx_free(&libuarte, p_evt->data.rxtx.p_data, p_evt->data.rxtx.length);
            break;
        }

Problem is that this approach doesn't work. If the .int_prio   = APP_IRQ_PRIORITY_LOW_MID then I can transfer a couple of chunks and the uart won't transmit anymore. Also, if the priority is higher then it won't ever send anything and program crashes.

The second thing I tried is to use memcpy to copy data into the second buffer and set the flag that data is received. Then in idle_state_handle function, I check if the flag is true and if it is I call ble_nus_data_send. That works really well up to a certain point. I tried various speeds and below ~45KBps I receive every byte in the app that I send. If I increase the speed to, for example, 80KBps then I start losing chunks of data. One test that I had is that I send 244bytes * 10000 and after the test is done I checked the app and I was missing around 100 packets. Also, this loss is not deterministic and varies between tests.

So, my question is what am I doing wrong? Is there another way to use libuarte async to transmit data at high speed than to poll that flag in the main loop? I know that the test board can transmit uart data at ~100KBps and I also managed to transmit test data with only NUS at also around 100KBps to the app. So BLE stack is good and connection from test board to nordic chip is also good so the problem must be with the way I use libuarte..

I would appreciate any input in how I can increase the speed while maintaining the lossless packet transfer.

Thanks,

Aleksa

[Edvin]

Hello Aleksa,

Your second approach seems reasonable. The first approach will work if things go slow, and you always have room in the softdevice queue, but that may not always be the case. Once ble_nus_data_send returns NRF_ERROR_RESOURCES, it will not be able to leave the interrupt, and hence not update the return value for the ble_nus_data_send().

So let us discuss the second approach:

If you receive a lot of uart messages, you will start filling the softdevice queue using ble_nus_data_send(). What happens if you call ble_nus_data_send, but it returns NRF_ERROR_RESOURCES, and then you get a new interrupt from the libuarte, which will overwrite the previous buffer that has not yet been successfully queued using ble_nus_data_send()? My assumption is that this happened. What should happen in this case? You can of course increase the size of the buffer, but if you keep receiving data faster than you are able to send it, the buffer will eventually fill up. 

Perhaps you need to look into how you can increase the throughput on the BLE link? You can experiment with the ble_app_att_mtu_throughput example to see what connection parameters that will give the highest throughput. I think it is something like: Conn_interval = 400ms, PHY: 2MBPS, MTU = max, DLE enabled, connection event length = connection interval. 

You can read the test description for this example here:

https://infocenter.nordicsemi.com/topic/sdk_nrf5_v17.0.0/ble_sdk_app_att_mtu.html

So first of all, you should use 2MBPS if possible. Then you should send as long packets as possible. This means that if you are sending a lot of small packets, try to combine them to one large before queuing it using ble_nus_data_send.

BR,

Edvin

[Aleksa]

Thank you for your reply. I think I have pretty good parameters in terms of BLE throughput. I had a test where I just went through for loop using ble_nus_data_send function and I was able to achieve more than 100KBps on conn_interval = 7.5, PHY: 2MBPS, MTU = max, DLE enabled. The only thing I still wasn't able to achieve was to increase conn_interval and make it work. But for our case, 100KBps is more than enough on the BLE part.

Now the BLE_NUS_MAX_DATA_LEN == 244 bytes and function logically assert if I try to send a larger length packet. So I made sure that the libuarte async buffer is also 244 bytes and I am sending 244 bytes packets from the test board to the nordic chip. 

In my mind, things should work like this. The test board transfers 244 bytes over UART. Libuarte receives the data and I get an event in which I set the flag that is caught in the main loop. Because everything is the same size there is no need for another queue or "gluing" few smaller packets into a larger one.

I am afraid that I may be approaching limits of this mechanism with libuarte and NUS which is troublesome. If that is the case do you maybe have any suggestions on how should I approach this problem to get the higher speed with reliable transfer above ~45KBps? I was also thinking about going level below and working with libuarte_drv library but I don't think that that is the correct way forward.

[Edvin]

I would check if it helps to double buffer the messages coming from your libuarte event handler, so that you alternate between two buffers. 

The first time you get a NRF_LIBUARTE_ASYNC_EVT_RX_DATA event, copy it to the first buffer, and the next time, copy it to the second buffer. And on each NRF_LIBUARTE_ASYNC_EVT_RX_DATA event, you trigger queuing to the ble_nus_data_send.

In that case, it will reduce the chance of two events coming in too close causing the buffer to be overwritten before it is sent to the softdevice. 

The softdevice has first priority, as you may know. So from time to time the NRF_LIBUARTE_ASYNC_EVT_RX_DATA events may be a bit delayed. By doubling the buffer you may have enough time to send the incoming packets between the connection events (every connection interval). 

[Aleksa]

Ok, so first few questions and then I'll show you what I tried.

1. What priority should I set .int_prio field in the nrf_libuarte_async_config_t struct so that it doesn't clash with BLE stack? All I know is that it should be higher than APP_IRQ_PRIORITY_LOWEST to not cause assert.
2. Do I need to use a do-while loop when calling ble_nus_data_send?
3. Do I need to do something in the BLE_GATTS_EVT_HVN_TX_COMPLETE event? If I am correct it should fire after ble_nus_data_send is finished.
4. What is the purpose of parameter _rx_buf_cnt in NRF_LIBUARTE_ASYNC_DEFINE? It says "Size impacts accepted latency
   * between NRF_LIBUARTE_ASYNC_EVT_RX_DATA event and
   * @ref nrf_libuarte_async_rx_free. But doesn't mention is it for better or for worse or why. In the libuarte example, the number of buffers is 3 but I tested with also 10 and didn't find any difference.

So here is what I figured out.

        case NRF_LIBUARTE_ASYNC_EVT_RX_DATA:
        {
            NRF_LOG_INFO("data length = %d", p_evt->data.rxtx.length);

            // measure received bytes
            cnt += p_evt->data.rxtx.length;

            if (active_buffer == 1)
            {
                buf = buffer2;
            }
            else if (active_buffer == 2)
            {
                buf = buffer1;
            }

            memcpy(buf, p_evt->data.rxtx.p_data, p_evt->data.rxtx.length);
            buf_len = p_evt->data.rxtx.length;

            err_code = ble_nus_data_send(&m_nus, buf, &buf_len, m_conn_handle);
            if ((err_code != NRF_ERROR_INVALID_STATE) &&
               (err_code != NRF_ERROR_RESOURCES) &&
               (err_code != NRF_ERROR_NOT_FOUND))
            {
                APP_ERROR_CHECK(err_code);
            }

            nrf_libuarte_async_rx_free(&libuarte, p_evt->data.rxtx.p_data, p_evt->data.rxtx.length);
            break;
        }

First I tried to put ble_nus_data_send in the do-while loop but it just straight-up hands. So I tried removing it from the loop and I got it to send data. The strange thing is that the original ble_app_uart example uses the loop and it works just fine.

I then conducted the following test. On the test board (STM32F4) I create a loop with 1000 counter and send 244 bytes with uart_send. I would expect to receive 244000 bytes on android application but that is not happening. I keep receiving less data then I should.

Now here is the catch. I verified that my test board sends the correct amount of data. Also, variable cnt in the code snippet is set to 244000 as it should be after the test. I assume that I did receive all data on the nordic board. I also know that when I call ble_nus_data_send in a for loop I get reliable transfer at ~100KBps. I can't really figure out what am I doing wrong.

If the counter is set to 244000 that means that I did enter the event exactly 1000 times and I should have called the function 1000 timer. Obviously that is not happening. Any help is appreciated.

[Aleksa]

I think I finally got it. I didn't really understand what you meant with buffering at first but I think I got it. Because there is more than 30KB or unused ram I split that into two buffers as you suggested and started filling them up in the NRF_LIBUARTE_ASYNC_EVT_RX_DATA event. After the buffer is full I set the flag in the main loop and switch the buffers so I can fill the next one. In the main loop, I bulk transfer the whole array instead of individual chunks like before.

I managed to transfer close to 100KB this way and received every byte in the android application. Naturally, there is more testing to be done with regards to what delay is acceptable in our case but I definitely think this is the way forward for higher speeds.

I just want to thank you again for timely responses and ideas it really helped.

Thanks,

Aleksa

[Edvin]

Aleksa said:

I think I finally got it. I didn't really understand what you meant with buffering at first but I think I got it. Because there is more than 30KB or unused ram I split that into two buffers as you suggested and started filling them up in the NRF_LIBUARTE_ASYNC_EVT_RX_DATA event. After the buffer is full I set the flag in the main loop and switch the buffers so I can fill the next one. In the main loop, I bulk transfer the whole array instead of individual chunks like before.

 Good! That was at least what I was trying to say. Since the softdevice may keep the CPU busy for short periods, you may receive two NRF_LIBUARTE_ASYNC_EVT_RX_DATA before you manage to handle the first one of them. So if you increase to having two buffers, the application should be able to push the data to the queue before the 3rd NRF_LIBUARTE_ASYNC_EVT_RX_DATA. At least as long as the BLE link has a higher throughput than the UART. 

Have a nice weekend.

Best regards,

Edvin