Zephyr uarte library missing UART_RX_BUF_RELEASED events

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

JinJie — Tue, 08 Feb 2022 02:44:07 GMT

Probably it is a race condition between uarte isr(ENDRX) and rx_timeout.

set short timeout and small rx dma ram buffer size in rx_enable(..., dma buf size, timeout) would make it easiler to cause race condition rapidly.

e.g. rx_enable(..., dmabuf_size = 20 byte, timeout = 5ms) and send plenty of bytes to uarte rx.

By the way, I use nrf52833 NCS on zephyr 2.6.0.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

JinJie — Mon, 07 Feb 2022 09:31:17 GMT

Anything update plz?

We met the same issue, and yet unable to determinate the reason.

Currently,

I found the normal function called chain when uarte rx receiving bytes:

isr(rx started) -> callback(new buffer rsp) -> isr(end rx) -> callback(buffer release)

PS: leave out the rx ready event cause it comes quiet frequently and randomly.

When we met this issue, we found isr(end rx) is not generate(missing endrx event or endrx interrupt mask), thus callback(buffer release) was not execute. i.e. isr(rx started) comes twice.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Kyrre Aalerud — Wed, 08 Sep 2021 07:32:34 GMT

I upploaded an image with the packets colorized around where the timing issue happens here: https://ibb.co/FxHkKTs

Each of the events are triggered in the driver which is running with 2ms on the timeout setting using 32 byte buffers to provoke it a bit.
Data is sent as bursts (using dma from other mcu) of single packets with 20ms interval. After the last long packet there is a pause of 4.5 seconds before some shorter responses are sent and the long packets start again. The short packets are sent with 50 and 90ms intervalls and should have come as seperate clean rx events but it is like some internal alignment prevents this?

Notice the RX timeout from my code discarding the end of the packet as it arrives 4.5s too late as part of next rx event triggered by the new shorter datapackets. Note the timestamps.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Susheel Nuguru — Wed, 08 Sep 2021 06:16:18 GMT

[quote user="Kyrre Aalerud"] if the transmission stops in the second buffer during this time (speculation) an rx even based on the timeout may not be generated.[/quote]

I think this should be the source of all the issue. If at any instance we can see that the timeout event is missing and did not fire at all, then that is a bug in the driver itself. The only issue before creating an internal ticket is that we need to prove it.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Kyrre Aalerud — Tue, 07 Sep 2021 12:33:11 GMT

For info, we have discovered some other anomaly with the rx events and the timeout system.

It appears that if a stream of data is received, spanning multiple rx buffers, depending on timing for processing the rx event caused by the first buffer end being reached, if the transmission stops in the second buffer during this time (speculation) an rx even based on the timeout may not be generated. Result is remaining data from the burst reception may sit in the second rx buffer and not generate any event. Only when a new burst of data (1 or more bytes) arrives later will the rx event be generated and it will also contain the missing data from the previous transmission.

When using a internal timeout to resynchronize packets this creates a very bad situation.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Susheel Nuguru — Thu, 12 Aug 2021 07:43:22 GMT

Thank you for this data. This is very useful for us to understand. Any chances that you can share your project that we can run on our DK, will make it easy for us to reproduce and debug. If we try to replicate it we might end up spending a lot of time searching for that timing critical event which triggers this.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Kyrre Aalerud — Thu, 12 Aug 2021 07:36:12 GMT

Very rarely I do get events where an odd number of bytes are received. My test uses 4 byte blocks and we use timeout in the driver to create rx events in a timely fashion at the same time as we have a timeout for parsing data in 4 byte blocks. When the odd number of bytes arrive the first bytes are blocks of 4 and valid data. The last byte is garbage and since it is alone within the rx event timeline the next rx event is a normal 4xN byte event again and arrives so late that our timeout is triggered. The incomplete packet is then discarded and the new incoming data is parsed in it's place. The data arriving are ACK's to commands sent to a STSPIN32 based custom brushless motor-controller and it is very important to have reliable communications to it. We have had to use both 2 layers of timing (1 to get timely data parsing, 1 to secure resynchronization of datastream) and packet validation to have this communications work. The length of serial traces are approx 15mm.

[01:32:09.484,832] <inf> hy5_uart: RX EVENT: rx.len=12 @ 5529484
[01:32:09.484,863] <err> hy5_uart: Buf FREE: 1 array: 1 0 0 1 1
[01:32:09.484,893] <err> hy5_uart: Buf REQ:  3 array: 1 1 0 1 1
[01:32:09.484,924] <inf> hy5_uart: ACK 134 @ 181190162
[01:32:09.484,924] <inf> hy5_uart: ACK 135 @ 181190162
[01:32:09.484,954] <inf> hy5_uart: ACK 136 @ 181190162
[01:32:09.488,922] <inf> hy5_uart: RX EVENT: rx.len=21 @ 5529488
[01:32:09.488,983] <inf> hy5_uart: ACK 137 @ 181190295
[01:32:09.488,983] <inf> hy5_uart: ACK 138 @ 181190295
[01:32:09.488,983] <inf> hy5_uart: ACK 139 @ 181190295
[01:32:09.489,013] <inf> hy5_uart: ACK 140 @ 181190295
[01:32:09.489,013] <inf> hy5_uart: ACK 141 @ 181190296
[01:32:09.688,323] <inf> hy5_uart: RX EVENT: rx.len=4 @ 5529688
[01:32:09.688,354] <err> hy5_uart: RX Timeout. 5529688-5529488=200. Discarded incomplete data of 1 byte.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Susheel Nuguru — Thu, 12 Aug 2021 07:06:00 GMT

for your reference, the internal ticket for the low level driver team to look at this is NCSDK-10801

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Susheel Nuguru — Thu, 12 Aug 2021 06:56:07 GMT

Thanks for the detailed explanation.

[quote user="Kyrre Aalerud"]My application checks the buffer reference and compares to it's list of buffers to identify the buffer number being freed.[/quote]

This above explanation from you tells me that this most likely is a race condition in the uart driver where it missed this released event totally. I will create an internal ticket for the bug.

It would be interesting info from you if you can confirm if the buffer with the missed released event had its buffer fully/partially filled? and also if the data in that buffer was relavant? Also would it be possible for you to give us a minimalistic project to reproduce this as this seems very timing related. This information will help the engineers focus on the code sections which are which cause this behavior.

Once again Thanks a lot for such a detail explanation and your experiments. But for now, since you are not doing dynamic allocation on request events and able to manage the round robin buffers in a good way (and also not losing data due to this bug) we hope that this bug will be fixed in the next releases.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Kyrre Aalerud — Wed, 11 Aug 2021 15:43:42 GMT

My application checks the buffer reference and compares to it's list of buffers to identify the buffer number being freed.
The buffers are allocated statically so these values/pointers never change.

Buf FREE x means it has identified the referenced buffer as buffer number x from 0 to 4.

It always responds to a buffer request with the next buffer in succession and starts the system by setting up the enable rx call using buffer 0. The Buf 1 request event has been logged inbetween something else during startup and thus was missing in the log but happened correctly.

Since I run a total of 5 buffers to debug this issue I see it usually always frees a buffer immediately once full, however this does not appear to always happen which leads me to believe the problem is either in the hardware or in the library during the handling of the events. I am thinking that there may exist a short timeframe where a flag may be set by hardware buy is cleared by software before or after the execution of the rx events callback etc.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Susheel Nuguru — Wed, 11 Aug 2021 12:54:00 GMT

This smells like a race condition but before that can you please explain to me what the number next to "Buf FREE X" means here? Is X something that is tracked by your application or you identify the index based on the buff pointer or you identify the buffer index freed based on some other logic?

In the above highlighted context, why does your application think that Buf Free is index 1 instead of index 0?

It looks like two buffer request can come in succession if second buffer is already given and is full before the application can process the UART_RX_RDY/UART_RX_BUF_RELEASED events. This might be something undocumented behavior but might not necessary translate to a missed event. It is possible that the Free event got postponed and your application index tracking is now confused due to two buffer requests coming in succession.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Kyrre Aalerud — Wed, 11 Aug 2021 10:06:35 GMT

Here is the same log with RX events as well. I had to compact it a bit to fit within 512k pastebin limits.
https://pastebin.com/15AJpMP9

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Kyrre Aalerud — Wed, 11 Aug 2021 09:42:46 GMT

The logs are very long before things happen so I had to use pastebin. Bottom of the log is the relevant part but all is included.

Log of REQ and FREE

The app keeps track of this array separately but it prints each event as they occur in the log.
It is clear that a free event does not happen. It does ask for two buffers in quick succession without freeing buffer 3 though, but frees 3 right after.

RE: Zephyr uarte library missing UART_RX_BUF_RELEASED events

Susheel Nuguru — Wed, 11 Aug 2021 06:46:06 GMT

Hi Kyrre,

Sounds very much like a race condition. I need to understand more about the chain of events that happened before digging into the code. Could you please help me understand with these questions.

Your application seems to be tracking which buffer is free and not instead of relying solely on these events. Is it possible that the application tracking logic might have a race condition? Could you rule that out?
Could you provide me the chain of UART events that are received by your application (a log file with a way to see EVENTS/buffer mapping would help).
What other application peripheral interrupts are enabled and being used when your application detected that a request arrived for a buffer that is not yet free?