ASYNC uart and UART_RX_RDY timeout

RE: ASYNC uart and UART_RX_RDY timeout

LongQi — Tue, 04 Jun 2024 03:19:42 GMT

Thanks, but interrupt occupy too much CPU time and my other thread will get affected. I use the data padding to fulfill the buffer like MrBurvall did for now.

RE: ASYNC uart and UART_RX_RDY timeout

MrBurvall — Mon, 03 Jun 2024 12:13:46 GMT

Hello, I reproduced it using two PCA10056 and the answer from Nordic in a private ticket was "Link here. The workaround as suggested in that case is to enable HW byte counting."

That works for me. No real fix from Nordic exists.

Best Regards, Markus

RE: ASYNC uart and UART_RX_RDY timeout

LongQi — Mon, 03 Jun 2024 11:24:53 GMT

any progress? I am using 2.6.1 and still have the issue.

RE: ASYNC uart and UART_RX_RDY timeout

MrBurvall — Thu, 07 Mar 2024 11:32:13 GMT

Hello mmm,

Where you able to find a solution for this problem?

I see the same behavior, workaround for me is to send some extra data, which is ignored. This triggers UART_RX_RDY.

Best Regards, Markus

RE: ASYNC uart and UART_RX_RDY timeout

Øivind — Thu, 02 Nov 2023 14:47:40 GMT

Hi, an update, we have received multiple tickets about this, so it appears that it may be a bug in the driver. I am currently looking into these reports and I'll keep you updated.

RE: ASYNC uart and UART_RX_RDY timeout

mmm — Thu, 26 Oct 2023 14:24:31 GMT

> From your log it seems that the single byte is not lost, as it is included at the start of the following message.

correct, this ticket is not about data getting lost, but UART_RX_RDY reporting an incorrect number of bytes received.

With 2.4.0 the number was kinda random, with 2.4.2 seems is it always under-reported by 1, but may be just a coincidence.

> Could you try debugging and check where this byte is when the RX RDY event is triggered

I do see the byte are all the 10 bytes in the buffer as expected; but UART_RX_RDY's evt->data.rx.len is wrong

RE: ASYNC uart and UART_RX_RDY timeout

Øivind — Thu, 26 Oct 2023 11:10:01 GMT

Hi,

From your log it seems that the single byte is not lost, as it is included at the start of the following message.

Could you try debugging and check where this byte is when the RX RDY event is triggered in your event handler? Check the contents of the RX buffer and whether data.rx.len matches the number of bytes in the RX buffer. A logic trace on the UART lines could also be insightful.

RE: ASYNC uart and UART_RX_RDY timeout

Einar Thorsrud — Wed, 25 Oct 2023 13:34:42 GMT

Hi,

Edvin is our of office so the case has been re-assigned. We will look into it and get back to you. Please let us know if you have had any updates on your end in the meantime.

Br,

Einar

RE: ASYNC uart and UART_RX_RDY timeout

mmm — Thu, 19 Oct 2023 11:20:04 GMT

Hi, long time, new update.

Im testing with toolchain 2.4.2 and seems better but still some issues.

Output and code from my test follow.
First column is time in seconds, if data is send/received and what is sent/received (received data has b'' around them as i print the raw bytes, to see if i get weird non-visible ascii bytes, for example \n is an actual line feed)

The test send a message every 2 seconds, the nrf is set to 100ms timeout, and send back data immediately.

1 byte is systematically not reported as received (tested with different length of data, seems always 1 is "lost"), the buffer swap happen correctly at 100 bytes, and all no data is lost.

1697713032.61: sent: 0123456789
1697713032.811: recv: b'INF: got 9 bytes\nDATA: -012345678-\n'
1697713034.618: sent: 0123456789
1697713034.818: recv: b'INF: got 9 bytes\nDATA: -901234567-\n'
1697713036.624: sent: 0123456789
1697713036.825: recv: b'INF: got 9 bytes\nDATA: -890123456-\n'
1697713038.631: sent: 0123456789
1697713038.832: recv: b'INF: got 9 bytes\nDATA: -789012345-\n'
1697713040.638: sent: 0123456789
1697713040.839: recv: b'INF: got 9 bytes\nDATA: -678901234-\n'
1697713042.645: sent: 0123456789
1697713042.846: recv: b'INF: got 9 bytes\nDATA: -567890123-\n'
1697713044.652: sent: 0123456789
1697713044.853: recv: b'INF: got 9 bytes\nDATA: -456789012-\n'
1697713046.66: sent: 0123456789
1697713046.861: recv: b'INF: got 9 bytes\nDATA: -345678901-\n'
1697713048.667: sent: 0123456789
1697713048.868: recv: b'INF: got 9 bytes\nDATA: -234567890-\n'
1697713050.674: sent: 0123456789
1697713050.774: recv: b'INF: detected end of buffer for 19 bytes\nDATA: -1234567890123456789-\n'
1697713052.681: sent: 0123456789
1697713052.882: recv: b'INF: got 9 bytes\nDATA: -012345678-\n'
1697713054.688: sent: 0123456789
1697713054.889: recv: b'INF: got 9 bytes\nDATA: -901234567-\n'

Code python:

import serial
import time

last_write = time.time()
with serial.Serial('/dev/ttyACM0', 921600, timeout=0.1) as ser:
 while True:
 x = ser.read(100) # read one byte
 if len(x):
 print(f"{round(time.time(), 3)}: recv: {x}")
 if time.time() - last_write >= 2:
 last_write = time.time()
 msg = "0123456789ABCDEFGHILMNOPQRSTUVZ"
 print(f"{round(time.time(), 3)}: sent: {msg}")
 ser.write(msg.encode("ascii"))
print("failed to open serial")

Code NRF:

#include <zephyr/types.h>
#include <zephyr/kernel.h>
#include <zephyr/drivers/uart.h>
#include <zephyr/usb/usb_device.h>

#include <zephyr/device.h>
#include <zephyr/devicetree.h>
#include <soc.h>

#include <stdio.h>

#define LOG_MODULE_NAME peripheral_uart
LOG_MODULE_REGISTER(LOG_MODULE_NAME);

uint8_t uart_buffer_rx[100];
volatile size_t uart_buffer_written = 0;
volatile size_t uart_buffer_completed = 0;
volatile bool disabled = true;

static const struct device *uart = DEVICE_DT_GET(DT_CHOSEN(nordic_nus_uart));

const int32_t UART_RX_TIMEOUT_US = 1000 * 100; // 100 millisecond timeout

void app_uart_async_callback(const struct device *uart_dev, struct uart_event *evt, void *user_data) {
 switch (evt->type) {
 case UART_TX_DONE:
 // do nothing
 break;
 
 case UART_RX_RDY:
 uart_buffer_written += evt->data.rx.len;
 break;
 
 case UART_RX_BUF_REQUEST:
 //assign same buffer; basically this is now a circular buffer
 uart_rx_buf_rsp(uart, uart_buffer_rx, sizeof(uart_buffer_rx));
 uart_buffer_completed++;
 uart_buffer_written = 0; 
 break;

 case UART_RX_BUF_RELEASED:
 break;

 case UART_RX_DISABLED:
 disabled = true;
 break;
 
 default:
 break;
 }
}

static int uart_init(void) {
 int err = 0;
 struct uart_config serial_config;
 uart_config_get(uart, &serial_config);
 uart_configure(uart, &serial_config);

 if (!device_is_ready(uart)) {
 return -ENODEV;
 }

 err |= uart_callback_set(uart, app_uart_async_callback, NULL);
 err |= uart_rx_enable(uart, uart_buffer_rx, sizeof(uart_buffer_rx), UART_RX_TIMEOUT_US);
 disabled = false;
 return err;
}

void log(const char *msg, ...) {
 static char s[1000];
 va_list argptr;
 va_start(argptr, msg);
 size_t len = vsnprintf(s, sizeof(s), msg, argptr);
 va_end(argptr);
 
 uart_tx(uart, s, len, SYS_FOREVER_MS);
 k_sleep(K_MSEC(1)); //92 char
}

void error(void) {
 while (true) {
 /* Spin for ever */
 k_sleep(K_MSEC(1000));
 }
}

int main(void) {
 int err = 0;

 err = uart_init();
 if (err) {
 log("ERR: uart_init");
 error();
 }

 log("started!");

 size_t last_uart_buffer_written = uart_buffer_written;
 size_t last_uart_buffer_completed = uart_buffer_completed;
 for (;;) {
 if (disabled){
 uart_rx_enable(uart, uart_buffer_rx, sizeof(uart_buffer_rx), UART_RX_TIMEOUT_US);
 if (err == 0)
 disabled = false;
 log("ERR: Serial re-enabled");
 }
 size_t number_of_oveflow = uart_buffer_completed - last_uart_buffer_completed;
 switch (number_of_oveflow){
 default: 
 {
 log("ERR: TOO MANY OVERFLOW FOR %d TIMES", number_of_oveflow);
 // we want to fallthrough to read whatever we got
 }
 case 1:
 {
 // now we print remaining buffer
 size_t remaining_data = sizeof(uart_buffer_rx) - last_uart_buffer_written;
 log("INF: detected end of buffer for %d bytes\n", remaining_data);
 log("DATA: -%.*s-\n", remaining_data, uart_buffer_rx+last_uart_buffer_written);
 // now reset buffer counters
 last_uart_buffer_completed += number_of_oveflow;
 last_uart_buffer_written = 0;
 // we want to fallthrough to read the initial data
 }
 case 0:
 {
 size_t available = uart_buffer_written - last_uart_buffer_written;
 if (!available){
 break;
 }
 log("INF: got %d bytes\n", available);
 log("DATA: -%.*s-\n", available, uart_buffer_rx+last_uart_buffer_written);
 last_uart_buffer_written += available;
 }
 }
 k_sleep(K_MSEC(10));
 }
}

RE: ASYNC uart and UART_RX_RDY timeout

Edvin — Fri, 11 Aug 2023 13:08:06 GMT

I understand. I am sorry for the late reply. I have been trying to clear up my backlog, because I will be out of office for a while.

Try adding RTT logging to your project, so that you can look into what the functions in the UART_TX_DONE event returns, and whether they work as you expected. I didn't manage to understand the logic flow during my last take on this ticket.

Check whether bytes_claimed have the correct value when UART_TX_DONE triggers, and whether k_sem_give(&tx_done) is called when it shouldn't be (if bytes_claimed != 0).

Since I will be out of office, I suggest you create a new ticket for follow up questions. You can refer to this one for background, but please upload your application there, if you changed anything at this point, and try to give a summary of your current status.

Best regards,

Edvin

RE: ASYNC uart and UART_RX_RDY timeout

mmm — Mon, 07 Aug 2023 13:13:34 GMT

Hi,

yes, those differences are desired.

This is a minimal example to reproduce and debug the issue, i did not want to add anything more than necessary.

I need to be able to saturate the 921600 baudrate, so i had to get rid of the interrupt implementation and switch to a DMA based. I managed to reach full write speed, but when i started to test the input speed, i notice the test failing and the issue was the weird timing.

Then i started to remove stuff, thinking it may be some issue with timing or concurrency..

Until i notice the issue is coming from the callback.

Note that the data is missing already in the callback event, so even if TX would misbehave, the issue is present; the TX output is just a nice to have

RE: ASYNC uart and UART_RX_RDY timeout

Edvin — Mon, 07 Aug 2023 13:01:46 GMT

Hello,

I see that your implementation differs quite a bit from the peripheral_uart sample. Is that on purpose? I played around with the sleep in your main() loops, and I see that it queues only parts of the message on every sleep cycle. Also, your UART_TX_DONE event (inside app_uart_async_callback()) is not successfully triggering the next byte, so this will continue to fill up the buffer, as you write to it.

I think you need to look into the UART_TX_DONE implementation that you are using. Especially the bytes_claimed handling. Is this working as you expect? Or are you freeing up the buffer faster than you are processing it? I also suggest adding logging via RTT, so that you can use LOG_INF() to see what's going on. (It is difficult to use printk, which is using the UART, when the UART is not working as you expect it to).

I see that when I remove all the printk() it is working closer to what I expect it to, but then again, as it says in the comments, these functions are not thread safe, so interrupting it with printk() may not be very fortunate.

Best regards,

Edvin

RE: ASYNC uart and UART_RX_RDY timeout

mmm — Fri, 04 Aug 2023 09:54:22 GMT

of course,here it is!
The board I am using is a FANSTEL BT480E
I started from a standard example and did a couple of configuration changes.

Note i am using

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y

devzone.nordicsemi.com/.../2480.peripheral_5F00_uart.zip

RE: ASYNC uart and UART_RX_RDY timeout

Edvin — Fri, 04 Aug 2023 06:56:15 GMT

I can't say that I have seen this before. Is it possible to zip the application that you are using to replicate this, so that I can give it a go? Are you running it on a DK, or are you using custom HW?

Best regards,

Edvin

RE: ASYNC uart and UART_RX_RDY timeout

mmm — Thu, 03 Aug 2023 14:21:59 GMT

Hi,

it does not fix the issue but I did indeed used a wrong example.
I am using 2.4.0, the example is bluetooth/peripherical_uart.

I changed this timeout to be 1 second and I still get timeout with partial received data:

I am sending, all together, 0123456789ABCDF, then wait for timeout, then send again the string 0123456789ABCDF

here what i see

[16:08:38:190] Hello world through the UART async driver␍␊

[16:08:47:780] UART_RX_RDY 15 536881776 0!␍␊

[16:08:47:780] RX 15: 0123456789ABCDE␍␊

[16:08:53:182] UART_RX_RDY 15 536881776 15!␍␊

[16:08:53:182] RX 15: F0123456789ABCD␍␊

at about 16:08:46 i send 0123456789ABCDF␍␊,

one second later we see

[16:08:47:780] UART_RX_RDY 15 536881776 0!␍␊

means the event has been called, with len 15 (wrong, should be 16), dma buffer at memory 536881776 (ok) and offset 0 (ok)

[16:08:47:780] RX 15: 0123456789ABCDE␍␊

is the main loop that send our data back, we see it send back the first 15 bytes, we are missing 'F'

Then we can see i dont do nothing for more than 5 seconds, and no timeout for the remaining data.

then at about 16:08:52 i send again 0123456789ABCDF

and we see

[16:08:53:182] UART_RX_RDY 15 536881776 15!␍␊

len 15 (wrong, should be 16 of the new message plus the 1 of the old message), dma buffer at memory 536881776 (ok) and offset 15 (ok, at least is aligned with what happen before)

[16:08:53:182] RX 15: F0123456789ABCD␍␊

we see that we finally obtained the last F, and the first 14 bytes, but now E and F are missing.. but the event UART_RX_RDY is never called...
My application has no way to know that there are 2 more bytes waiting.

full test up until buffer swap:

[16:08:38:190] UART Async example started␍␊

[16:08:38:190] UART_RX_BUF_REQUEST␍␊

[16:08:38:190] Hello world through the UART async driver␍␊

>>> 0123456789ABCDEF

[16:08:47:780] UART_RX_RDY 15 536881776 0!␍␊

[16:08:47:780] RX 15: 0123456789ABCDE␍␊

>>> 0123456789ABCDEF

[16:08:53:182] UART_RX_RDY 15 536881776 15!␍␊

[16:08:53:182] RX 15: F0123456789ABCD␍␊

>>> 0000000000

[16:12:37:011] UART_RX_RDY 9 536881776 30!␍␊

[16:12:37:011] RX 9: EF0000000␍␊

>>> 1111111111

[16:12:49:414] UART_RX_RDY 9 536881776 39!␍␊

[16:12:49:414] RX 9: 000111111␍␊

>>> 3333333333

[16:12:55:008] UART_RX_RDY 9 536881776 48!␍␊

[16:12:55:008] RX 9: 111133333␍␊

>>> 4444444444

[16:12:57:806] UART_RX_RDY 10 536881776 57!␍␊

[16:12:57:806] RX 10: 3333344444␍␊

>>> 5555555555

[16:12:59:803] UART_RX_RDY 9 536881776 67!␍␊

[16:12:59:819] RX 9: 444444555␍␊

>>> 6666666666

[16:13:02:616] UART_RX_RDY 9 536881776 76!␍␊

[16:13:02:616] RX 9: 555555566␍␊

>>> 7777777777

[16:13:03:545] UART_RX_RDY 15 536881776 85!␍␊

[16:13:03:545] UART_RX_BUF_RELEASED␍␊

[16:13:03:545] UART_RX_BUF_REQUEST␍␊

[16:13:03:545] RX 15: 666666667777777␍␊

>>> 0123456789ABCDEF
[16:14:08:549] UART_RX_RDY 14 536881876 0!␍␊

[16:14:08:549] RX 14: 7770123456789A␍␊

we see the buffer swap happen correctly at len 100 and no data is lost and the second buffer is used, but we always have few byes lost, and the amount of lost byte is not constant.
It can become so bad, sometimes i can be more than 10 bytes late!

RE: ASYNC uart and UART_RX_RDY timeout

Edvin — Thu, 03 Aug 2023 07:45:42 GMT

Hello,

What NCS version are you using? The reason I ask is that we have a bug in some of the previous ncs versions (fixed in v2.4.0) where the uart_rx_enable() expects a timeout in microseconds, but we provided it in milliseconds. So depending on what your UART_RX_TIMEOUT_MS is set to, it may only receive 1/1000 of what you expect it to be.

If that is the issue, the fix is easy. Just increase the UART_RX_TIMEOUT_MS by a factor of 1000, and you should be fine.

Let me know if that doesn't help.

Best regards,

Edvin