Uart: wrong offset after buffer overflow

RE: Uart: wrong offset after buffer overflow

papatel — Wed, 19 Jun 2024 08:32:21 GMT

I cannot express how grateful that this thread exists and I have stumbled on it. We have a nrf52 design that interfaces with a nrf91. We observed minor uart corruption at 115200 baud. This was a red flag because I've done enough designs to know that at a trace length of 16mm for TX and RX, there should be ZERO uart corruption if both sides are implemented correctly.

We spent 2+ weeks of debugging and refactoring asynchronous uart reception code on both the nrf52 and the nrf91. We also spent serious resources debugging our PCB for potential noise issues being injected into the uart lines. Nothing fruitful from those efforts.

Finally today we realized that turning off BLE advertisements yielded significantly reduced uart corruption but we sill saw corruption at higher baud rates (which should be easily electrically supported).

And today I found this gem of a thread and we are running at 1MBPS uart with megabytes of tested CRC32 checksum data with not a single bit error:

CONFIG_SERIAL=y
CONFIG_UART_ASYNC_API=y
CONFIG_UART_0_NRF_HW_ASYNC=y
CONFIG_UART_0_NRF_HW_ASYNC_TIMER=2

Friendly feedback to Nordic: Pretty please make this concern front and center in the UART tutorial. Or better yet, make hardware byte counting the default for all projects you can. And where you can't assign the timer or PPI instance because they are used/unavailable, explicitly warn users data transfers over UART may be unreliable.

RE: Uart: wrong offset after buffer overflow

MaLu — Wed, 13 Apr 2022 06:23:38 GMT

Hello,
Well, spending too much time in the interrupt handler was very much a rookie mistake on my part. Good to know though, that "excess data" doesn't just get lost, but overwrites existing data by design, in case the new buffer does not get given on time. Definitely something to keep in mind.

Of course I adjusted my test FW and it now works fine.

Thank you all so much!

RE: Uart: wrong offset after buffer overflow

Krzysztof Chruscinski — Tue, 12 Apr 2022 09:19:56 GMT

So hardfault is gone on 1.9.1? That's good.

Regarding unexpected behavior with data being overwritten. It is a feature not a bug. There is a DMA linking setup between the buffers, meaning that ENDRX event triggers STARTRX. This works well as long as you handle next buffer request before current transfer ends.

So basically, to have reliable reception you must provide new buffer on time. By increasing buffer size you increase that accepted latency. Unfortunately, there is no means in the driver to detect that you didn't provide the buffer on time. In your case, you see this issue because you are printk from that interrupt handler which blocks interrupt for significant amount of time. If you would store somewhere info and print it from thread context issue would go away.

RE: Uart: wrong offset after buffer overflow

MaLu — Tue, 12 Apr 2022 05:17:02 GMT

The second issue (which I cannot edit the reply of for some reason), I tested with NCS 1.8.0.

The hard fault seems to be gone in 1.9.1, but some strange behaviour still persists.

Here I alternated pasting lower and upper case alphabet, so beginning with

abcdefghijklmnopqrstuvwxyz then ABCDEFGHIJKLMNOPQRSTUVWXYZ then lower case again, until the uart stopped with seemingly no error.

As you can well see, thedata that is (seemingly) received is not in line with what I sent to the uart1.

As I said, I encourage you to experiment eith this as well. If I should open another issue for this, please tell me so and I will.

RE: Uart: wrong offset after buffer overflow

MaLu — Tue, 12 Apr 2022 04:50:14 GMT

Hello, sorry for the late answer.

This seems to indeed have fixed the issue of the wrong offset, thank you a TON! Unfortunately I have discovered another issue, which might be known/intended/tolerated, but it sure isn't intuitive for me.

The Uart async driver has DMA buffers, which are to be provided. if you fill up both those buffers at once, the FW just hard faults. I again have made a video and linked a FW. please also look into this.

www.youtube.com/watch

devzone.nordicsemi.com/.../hello_5F00_world_5F00_UART_5F00_rx_5F00_test_5F00_2.zip

PS: note for the video: I have 32 byte for the example where I paste the numbers. it obviously also crashes when you use more than that. you dont have to fill them up exactly. Please try it out with divfferent lengths, to find strange behaviour.

RE: Uart: wrong offset after buffer overflow

Krzysztof Chruscinski — Fri, 08 Apr 2022 05:16:16 GMT

I think that the issue is with driver configuration and rx byte counting. By default software tries to count bytes but it fails when interrupt is not handled on time. That results in missing rx events (eventually you will get event on the buffer boundary). Alternative is to use hw byte counting but that requires kconfig setup and dedicating one TIMER peripheral instance like in the example below where TIMER2 is used for that:

CONFIG_UART_1_NRF_HW_ASYNC=y
CONFIG_UART_1_NRF_HW_ASYNC_TIMER=2

github.com/.../nrf9160dk_nrf9160.conf

RE: Uart: wrong offset after buffer overflow

MaLu — Thu, 07 Apr 2022 09:56:49 GMT

Hi,

I gotta be a hassle again. Is this still looked at? If you need more info just say it. As mentioned melow, other people seem to have similar issues. UART Async is unusable in this state.

RE: Uart: wrong offset after buffer overflow

MaLu — Fri, 01 Apr 2022 05:09:41 GMT

just pasting a link here to another thread with the same issue. Maybe their explenation will get the case solved quicker
devzone.nordicsemi.com/.../nrf-connect-sdk-async-uart-rx-loss

RE: Uart: wrong offset after buffer overflow

MaLu — Mon, 28 Mar 2022 12:27:03 GMT

Hey,
how's it going?

RE: Uart: wrong offset after buffer overflow

Øyvind — Tue, 15 Mar 2022 10:05:28 GMT

Hi, I tried to debug looking at the video you provided and have not found the reason.

Still working on this. Apologies for the time this is taking.

-Øyvind

RE: Uart: wrong offset after buffer overflow

MaLu — Tue, 15 Mar 2022 06:27:06 GMT

Hi,
I am once again asking for a status update, as I need to inform my team as well.

RE: Uart: wrong offset after buffer overflow

MaLu — Fri, 11 Mar 2022 05:56:07 GMT

I made a video anyway to help reproduce the behaviour.

www.youtube.com/watch

RE: Uart: wrong offset after buffer overflow

MaLu — Wed, 09 Mar 2022 13:40:18 GMT

Hi,
thanks for the feedback so far.
the memcpy you mentioned is honestly just for the visualization.

when you debug into the rx event and look at evt->data.rx.buf in memory, you can see, that in the case of the overflow, new bytes are added to a different offset than evt->data.rx.offset reports

I mainly copied the evt->data.rx.buf to au8Main_DummyBuf to easily watch it in a debugger watch window.

Were you able to replicate the issue of new chars arriving into evt->data.rx.buf on a different offset as evt->data.rx.offset ? If not I will make a video or something to help replicate the case

RE: Uart: wrong offset after buffer overflow

Øyvind — Wed, 09 Mar 2022 13:19:06 GMT

*** Booting Zephyr OS build v2.7.99-ncs1  ***
MAIN START!
uart rx buf req
evt len = 2, evt offs = 0, actual data in 0
12

evt len = 14, evt offs = 2, actual data in 2
3456789012345


freeing buffer
uart rx buf req
evt len = 2, evt offs = 0, actual data in 2
12

evt len = 14, evt offs = 2, actual data in 2
3456789012345


freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256
1

evt len = 15, evt offs = 1, actual data in 256
23456789012345


freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256
1

evt len = 15, evt offs = 1, actual data in 256
23456789012345


freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256
1

evt len = 15, evt offs = 1, actual data in 256
23456789012345


freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256
1

evt len = 15, evt offs = 1, actual data in 256
23456789012345


freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256
1

evt len = 15, evt offs = 1, actual data in 256
23456789012345


freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256
1

evt len = 15, evt offs = 1, actual data in 256
23456789012345


freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256
1

evt len = 15, evt offs = 1, actual data in 256
23456789012345


freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256
1

evt len = 15, evt offs = 1, actual data in 256
23456789012345


freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256
1

evt len = 15, evt offs = 1, actual data in 256
23456789012345


freeing buffer
uart rx buf req
evt len = 2, evt offs = 0, actual data in 256
12

evt len = 14, evt offs = 2, actual data in 256
3456789012345


freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 0



evt len = 1, evt offs = 1, actual data in 1
1

evt len = 14, evt offs = 2, actual data in 3
23456789012345

freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 2



evt len = 15, evt offs = 1, actual data in 2
123456789012345

freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256



evt len = 15, evt offs = 1, actual data in 256
123456789012345

freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256



evt len = 15, evt offs = 1, actual data in 256
123456789012345

freeing buffer
uart rx buf req
evt len = 2, evt offs = 0, actual data in 256

1

evt len = 14, evt offs = 2, actual data in 256
23456789012345

freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256



evt len = 15, evt offs = 1, actual data in 256
123456789012345

freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256



evt len = 15, evt offs = 1, actual data in 256
123456789012345

freeing buffer
uart rx buf req
evt len = 2, evt offs = 0, actual data in 256

1

evt len = 14, evt offs = 2, actual data in 256
23456789012345

freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256



evt len = 15, evt offs = 1, actual data in 256
123456789012345

freeing buffer
uart rx buf req
evt len = 1, evt offs = 0, actual data in 256



evt len = 15, evt offs = 1, actual data in 256
123456789012345

freeing buffer
uart rx buf req

There is clearly something going on.

Looks like the root cause is the function memcpy copies the whole buffer. There is definitely something off.

memcpy(au8Main_DummyBuf, evt->data.rx.buf, MAIN_UART_BUF_LEN);

Still trying to look more into the root cause.

-Øyvind

RE: Uart: wrong offset after buffer overflow

Øyvind — Wed, 09 Mar 2022 08:29:21 GMT

My sincere apologies for not responding to this earlier. I will provide an answer to you by end of today.

Sorry for the inconvenience.

RE: Uart: wrong offset after buffer overflow

MaLu — Tue, 08 Mar 2022 09:49:15 GMT

Hi,
hate to be a hassle, but I just wanted to ask, if you had a quick chance to look into this yet. I need this to work for my project at work, so I wanted to get a bit of an update.

Thanks in advance!

RE: Uart: wrong offset after buffer overflow

MaLu — Mon, 07 Mar 2022 06:51:28 GMT

Hi,

Thanks for the update. I tried to closely follow the lpuart example but with a regular uart interface, as it was also using an async api.

RE: Uart: wrong offset after buffer overflow

Øyvind — Wed, 02 Mar 2022 12:01:06 GMT

Hello,

I'm sorry but I have not been able to look into your case yet. Thanks for verifying the issue with both 1.8.0 and 1.9.0. Have you look at the samples we have specific to nRF9160?

RE: Uart: wrong offset after buffer overflow

MaLu — Mon, 28 Feb 2022 05:43:21 GMT

Hi,
thank you for looking over it.

I am using SDK 1.8.0

Edit: I have just tested my sample program and it shows the same behaviour with SDK 1.9.0

RE: Uart: wrong offset after buffer overflow

Øyvind — Fri, 25 Feb 2022 14:04:22 GMT

Hello,

I will have a look at the provided project. What version of the nRF Connect SDK are you running?

Kind regards,

Øyvind