Libuarte_async bug(s) - missing data / wrong buffer returned.

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

thefool — Wed, 12 Feb 2020 13:44:21 GMT

Yes, this is quite possible. While testing, we did see some cases where the data was transferred in smaller blocks:

Rx 17 -> 955
Rx 13 -> 972
Rx 16 -> 985
Rx 16 -> 1001
Rx 7 -> 1017
Rx 6 -> 0
Rx 20 -> 6
Rx 22 -> 26
Rx 15 -> 48
Rx 21 -> 63
Rx 17 -> 84
Rx 13 -> 101
Rx 14 -> 114
Rx 11 -> 128
Rx 13 -> 139
Rx 13 -> 152
Rx 13 -> 165
Rx 15 -> 178
Rx 25 -> 193
Rx 13 -> 218
Rx 13 -> 231
Rx 12 -> 244
Rx 3 -> 256
Rx 18 -> 259
Rx 17 -> 277
Rx 16 -> 294
Rx 17 -> 310
Rx 15 -> 327

Even though one packet that we transmit is 255 bytes.
We are transmitting everything available using DMA on the STM as well, so this case only occurs while our firmware is filling the TX buffer at exactly the same time that DMA is transferring the data. Since we fill the buffer with data we are getting from USB, it is possible that there interrupts are delayed, and there are pauses between those small 3-25 byte chunks of data, and there is a chance they're more than 100us.

So yes, it's quite possible that there are almost back to back timeouts of 100us, but I think that libuarte should be able to handle that too, even if both back to back timeout handlers are delayed.
Lets just take the first three chunks, and say there were 150us delay in between each pair, and that the timeout handler only started in the middle of the 16 byte chunk.

I think the counter should just keep counting, so when the handler reads it, it would be at 17+13+8=28 bytes. This is what it should tell us in the callback, we copy that data, and tell it to free it. So it moves the pointer by 28 bytes. Immediately following the second handler is called, and reads the counter which is still at 28 or maybe 29 bytes by now, so it returns 0 or 1 byte, then frees that 1 byte and exits.

I don't see a reason why the second interrupt handler should somehow fail...

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

Kenneth — Wed, 12 Feb 2020 13:15:37 GMT

Hi,

If my assumption should be any way correct, it would require that the incoming data from the STM32 is not back to back, but may actually have sporadic delay's of tens of us between bytes. If that is the case I think it could be that you will have two timeouts of 100us almost back to back, do you have any indication this may be the case by looking at the data from the STM32?

Kenneth

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

thefool — Wed, 12 Feb 2020 12:03:15 GMT

[quote userid="2111" url="~/f/nordic-q-a/57431/libuarte_async-bug-s---missing-data-wrong-buffer-returned/233140"]My only input is that I would recommend to increase .timeout_us = 100 to 300 (or possible more for testing). Reasoning is that depending on other interrupts that may be occurring (causing the uart to be pending), you want to make sure that you can always handle the uart interrupt in time before the next may occur.[/quote]

So, after more testing, this did indeed solve or work around our issue in the main project. We will set it to 1ms to be safe.

However, we fail to understand exactly why this is causing a problem, and think this should be able to be fixed in your libuarte implementation. I don't see a reason why a delayed interrupt handler should cause libuarte to give us old data, or why an interrupt would need to be handled before the next could occur. After all, these are not uart interrupts, but timer/counter interrupts,right? None of these should do anything to uart, which should happily continue receiving.

To be honest, I did not fully understand the implementation of libuarte, as I have not spent enough time with PPI to really konw exactly what you guys are doing there, and there are very few comments, so I can't see exactly where things are going wrong, but I'm assuming these things should be your design goals:

The counter that counts incoming bytes should not be stopped by the timeout, but only be reset by the buffer full event, or actually it should just reset whenever it reaches 128 (or whatever the buffer size is)
The timeout interrupt handler should just read the counter, check how much was copied last time, and tell us where and how much is new in the buffer.
The buffer full interrupt should do exactly the same, except it doesn't need to read the counter. Also, it should prepare the next buffer to write to.

My assumption is that things start going wrong if we're getting several timeout interrupts, as well as a buffer full interrupt before the first one can be handled, and that this causes one of the handlers to read data from the wrong buffer.

As far as we can see, the data we're seeing in a case like this is several hundred bytes old, so it seems like the counter is read correctly, but it's reading from a new a previously used buffer. (if it were using the wrong counter value, and reading from the right buffer, the data would be less than 128 bytes old).

Basically my question to Nordic is: Does libuarte handle the case correclty when one or more timeout interrupts and buffer full interrupts are delayed by the softdevice, and in the worst case the buffer full interrupt is handled before the timeout interrupt?

And that last part poses the question: if two interrupts have the same priority, is it guaranteed that they are handled in the same order they occured, or do peripherals in the same priority group have an internal priority (eg timer0 always being handled before timer1)? Does the DMA buffer full have a higher priority than the timeout interrupt?

My worry is that our workaround with setting such a long timeout only makes this case much more seldom, so we're not seeing it during testing, but that it might still occur at some point after all.

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

Krzysztof Chruscinski — Tue, 11 Feb 2020 13:10:59 GMT

Hi,

I've tried your example and after tweaking it a bit i see no errors. I've made following changes:

- commented out printf("\r\nUART started.\r\n"); i didn't check if it is the case but if retarget goes to uart then it will collide with libuarte driving same uart.

- moved initialization of libuarte after advertising init and adding 1second delay before start. This is to ensure that both boards are up and running before TX is started. If RX is started when TX is ongoing then there is no possibility to synchronize. RX is started in the middle of a byte and received data is garbled

- changed initilization of last_received_byte to (TX_SIZE- 1) to ensure that first byte is not received as error.

Source file attached.

Krzysztof

devzone.nordicsemi.com/.../3364.main.c

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

Gasper — Tue, 11 Feb 2020 10:16:57 GMT

Sure, I can make the trace, although that will not help a thing. The low-level UART errors (framing) make sense, they only appear when the libuarte is being initialized/enabled, so a few of those at the beginning isn't the problem. The problem is the data itself, for which the errors are reported when the buffer fills out, at which point the data is checked and copied to a queue for ble transmission, and then the libuarte buffer is freed.

Anyway, here is the trace, together with the test project that generated the trace: devzone.nordicsemi.com/.../trace.tar.gz

The project is tested on two nrf52840_DK boards, so you could test it as well, might be easier than this back and forth.

On the other hand, increasing the idle timeout on our main project helped the issue, although I still feel it shouldn't be an issue in the first place, at least not like that. I would understand missing bytes if callback handling took to long (in the real project we don't parse the data in the callback handler, we just copy it to our main buffer queue).

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

Kenneth — Tue, 11 Feb 2020 08:30:21 GMT

[quote user="Gasper"]You can find a logic analyzer trace in the csv file: 8473.rx.tar.gz. [/quote]

I do not see any errors in this trace?

This doesn't really display what I am looking for. I thought maybe you could make a logic analyzer trace (e.g using https://www.saleae.com/) where you did a trace of the UART RX pin and toggled a IO on UART errors, this would make it easier to go to the where the issue occurred and look at the actual timing on a bit level for previous bytes (if you were able to also output on UART TX the expected value and received value, then we could really see on a bit level for the previous UART RX what may have happened). The .csv is not detailed enough for this.

How frequently does an error occur?

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

Gasper — Mon, 10 Feb 2020 12:13:48 GMT

Enabling hfclk doesn't change anything.

You can find a logic analyzer trace in the csv file: devzone.nordicsemi.com/.../8473.rx.tar.gz. I've truncated the trace at the beginning to where Rx started - the nrf_libuarte_async_enable call returned in the middle of the first 0x62 byte, which is probably the framing error reported at the beginning. The first full buffer is reported (NRF_LIBUARTE_ASYNC_EVT_RX_DATA) after ~2.826ms at timestamp 2.10230902. The log for this trace is:

<warning> app: Boot
<warning> app: Test TX started.
<info> app: Debug logging for UART over RTT started.
<error> app: NRF_LIBUARTE_ASYNC_EVT_ERROR errorSrc 4
<info> app: Rx 128@x20004B84
<error> app: RX: expected x65, got x89 instead
<error> app: RX: expected x8A, got x8D instead
<error> app: RX: expected x8E, got x91 instead
<error> app: RX: expected x92, got x95 instead
<error> app: RX: expected x96, got x99 instead
<error> app: RX: expected x9A, got x9D instead
<error> app: RX: expected x03, got xA1 instead
<error> app: RX: expected x07, got xA5 instead
<error> app: RX: expected x0B, got xA9 instead
<error> app: RX: expected x0F, got xAD instead
<error> app: RX: expected x13, got xB1 instead
<error> app: RX: expected x17, got xB5 instead
<error> app: RX: expected x1B, got xB9 instead
<error> app: RX: expected x1F, got xBD instead
<error> app: RX: expected x23, got xC1 instead
<error> app: RX: expected x27, got xC5 instead
<error> app: RX: expected x2B, got xC9 instead
<error> app: RX: expected x2F, got xCD instead
<error> app: RX: expected x33, got xD1 instead
<error> app: RX: expected x37, got xD5 instead
<error> app: RX: expected x3B, got xD9 instead
<error> app: RX: expected x3F, got xDD instead
<error> app: RX: expected x43, got xE1 instead
<error> app: RX: expected x47, got xE5 instead
<error> app: RX: expected x4B, got xE9 instead
<error> app: RX: expected x4F, got xED instead
<error> app: RX: expected x53, got xF1 instead
<error> app: RX: expected x57, got xF5 instead
<error> app: RX: expected x5B, got xF9 instead
<error> app: RX: expected x5F, got xFD instead
<error> app: RX: expected x63, got x81 instead
<error> app: Errors: 31 -> when got 128 @ x20004B84
<info> app: Rx 128@x20004C04

Tx started roughly 20us after Rx. Spacing between packets is roughly 10us. Baudrate is set in fw to 460800. Tx packet size is 155 bytes (increasing from 0x00 to 0x9A), Rx buffer size is 128 and the size mismatch is intentional as in the original project we have variable packet lengths. Since this is a data stream (for our use case as well), we don't particularly care that packet end isn't detected in this case, but we do need it (and it works fine) for a different mode of operation in our core project.

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

Kenneth — Mon, 10 Feb 2020 08:57:09 GMT

Can you check if starting the HFCLK may help? E.g. nrf_drv_clock_hfclk_request()/sd_clock_hfclk_request().

Do you have a logic analyzer trace of the serial incomming data? I just want to check the size of the packets, the delay between packets/bytes and baud rate.

[quote user="Gasper"]As for the errorSrc 0 thing, I found the error, not sure how I've missed it before: in libuarte_async.c in the handler for NRF_LIBUARTE_DRV_EVT_ERROR, you didn't actually copy the errorSrc.[/quote]

Thanks, I will report it.

Best regards,
Kenneth

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

Gasper — Fri, 07 Feb 2020 14:58:11 GMT

Hi, We're using the latest SDK (16.0.0), but we first ported to 15.0 because that's what the project is using. I have also increased timeout_us to 500 for testing, but it didn't change a thing. We're still seeing a bunch of wrong incoming data, usually in the first RX event. I have confirmed that the other side is sending a continuous stream of bytes from 0x00 - 0x9A, then it resets and starts with 0x00 again. This doesn't make sense, since it reports values above 0x9A, which are not being sent over the line at all:

<warning> app: Boot
<error> app: NRF_LIBUARTE_ASYNC_EVT_ERROR errorSrc 4
<error> app: NRF_LIBUARTE_ASYNC_EVT_ERROR errorSrc 4
<warning> app: Test TX started.
<info> app: Debug logging for UART over RTT started.
<error> app: NRF_LIBUARTE_ASYNC_EVT_ERROR errorSrc 4
<info> app: Rx 128@x20004B84
<error> app: RX: expected x65, got xAB instead
<error> app: RX: expected x11, got xD5 instead
<error> app: RX: expected x3B, got xE5 instead
<error> app: RX: expected x4B, got xF5 instead
<error> app: RX: expected x5B, got x85 instead
<error> app: RX: expected x86, got x89 instead
<error> app: RX: expected x8A, got x8D instead
<error> app: RX: expected x8E, got x91 instead
<error> app: RX: expected x92, got x95 instead
<error> app: RX: expected x96, got x99 instead
<error> app: RX: expected x9A, got x9D instead
<error> app: RX: expected x03, got xA1 instead
<error> app: RX: expected x07, got xA5 instead
<error> app: RX: expected x0B, got xA9 instead
<error> app: RX: expected x0F, got xAD instead
<error> app: RX: expected x13, got xB1 instead
<error> app: RX: expected x17, got xB5 instead
<error> app: RX: expected x1B, got xB9 instead
<error> app: RX: expected x1F, got xBD instead
<error> app: RX: expected x23, got xC1 instead
<error> app: RX: expected x27, got xC5 instead
<error> app: RX: expected x2B, got xC9 instead
<error> app: RX: expected x2F, got xCD instead
<error> app: RX: expected x33, got xD1 instead
<error> app: RX: expected x37, got xD5 instead
<error> app: RX: expected x3B, got xD9 instead
<error> app: RX: expected x3F, got xDD instead
<error> app: RX: expected x43, got xE1 instead
<error> app: RX: expected x47, got xE5 instead
<error> app: RX: expected x4B, got xE9 instead
<error> app: RX: expected x4F, got xED instead
<error> app: RX: expected x53, got xF1 instead
<error> app: RX: expected x57, got xF5 instead
<error> app: RX: expected x5B, got xF9 instead
<error> app: RX: expected x5F, got xFD instead
<error> app: RX: expected x63, got x81 instead
<error> app: Errors: 36 -> when got 128 @ x20004B84
<info> app: Rx 128@x20004C04
<info> app: Rx 128@x20004C84

As for the errorSrc 0 thing, I found the error, not sure how I've missed it before: in libuarte_async.c in the handler for NRF_LIBUARTE_DRV_EVT_ERROR, you didn't actually copy the errorSrc. Basically, the .data portion is missing:

        nrf_libuarte_async_evt_t evt = {
            .type = NRF_LIBUARTE_ASYNC_EVT_ERROR,
            .data = {
                .errorsrc = p_evt->data.errorsrc
            }
        };

Now that I've added that I get the errorsrc4 == framing error, which makes sense, since the other side is already sending while this side is initializing, meaning RX line is changing, and so we get framing errors (a few, right after initialization).

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

thefool — Fri, 07 Feb 2020 10:18:14 GMT

I'm working with Gasper on this, and when looking at the libuarte implementation, I can't help but wonder if the byte counter/timer is really the best way to handle getting the amount of bytes received. DMA HAS to know where it should write the next byte it gets from UART, so I'd assume there must be a way to somehow get that information. Whether it's a register or a memory location...

ie when the timeout fires, we fetch that information (eg by subtracting the current buffer start pointer from the current dma pointer, which should give us the number of bytes received).

would this not be possible and save resources (and potential error sources)?

RE: Libuarte_async bug(s) - missing data / wrong buffer returned.

Kenneth — Thu, 06 Feb 2020 18:41:15 GMT

I am not aware of any issue, but using the latest is in general the recommended.

My only input is that I would recommend to increase .timeout_us = 100 to 300 (or possible more for testing). Reasoning is that depending on other interrupts that may be occurring (causing the uart to be pending), you want to make sure that you can always handle the uart interrupt in time before the next may occur.

"errorSrc 0" was very odd, maybe you can set a breakpoint on the error and double check value. Making sure RXD is always pulled up can be something to check.

Best regards,
Kenneth