Zephyr UART Async receive drops a single character

RE: Zephyr UART Async receive drops a single character

Hieu — Thu, 29 Aug 2024 10:11:47 GMT

Yes, I ported my test app from 2.2.0 to 2.6.1 yesterday, and also see that it works correctly now.

If you were concerned about risks or effort costs, you could first try just porting just your UART functionality to a standalone 2.6.1 project to test the water first, before committing to upgrade the entire project.

RE: Zephyr UART Async receive drops a single character

KjellArne Amina — Thu, 29 Aug 2024 08:07:55 GMT

Thanks for the quick feedback. I saw Jacobs post about the issue being present in 2.5.1 so wasn't too optimistic. We're still on 2.4.1, sorry for not mentioning that.

I guess this means we have one more reason to upgrade.

RE: Zephyr UART Async receive drops a single character

Hieu — Wed, 28 Aug 2024 14:38:25 GMT

Thanks, Thilo, for the confirmation.

@KjellArne Amina could you please share what version of the SDK you are using, and if 2.6.1 works for you?

RE: Zephyr UART Async receive drops a single character

C.Thilo — Wed, 28 Aug 2024 11:55:29 GMT

For us, this problem is fixed with nrf sdk 2.6.1. We upgrade from 2.4.3 to 2.6.1, so might be fixed in a previous version between theses two.

RE: Zephyr UART Async receive drops a single character

KjellArne Amina — Wed, 28 Aug 2024 11:52:06 GMT

We recently ran into this issue ourselves. Has it been fixed in a recent SDK release?

RE: Zephyr UART Async receive drops a single character

Hieu — Fri, 22 Mar 2024 14:45:14 GMT

Hi JacobPrestonSynapse,

Are you able to reproduce the issue on a simple sample, such as the Echo sample, with just some light modifications? That would help us a lot with investigating the issue.

Hieu

Please be informed that our team is currently short-handed, and there will be delay in processing this request.
The situation will improve after the Easter holiday.
Our apologies for the inconvenience.

RE: Zephyr UART Async receive drops a single character

JacobPrestonSynapse — Mon, 18 Mar 2024 16:22:46 GMT

Just chiming in here that we've been seeing this issue on an nRF52840 with SDK v2.5.1. We are using the ASYNC UART with HW timers enabled. I'm not sure what more information I can provide that would be helpful at this point as other users have already posted, but we don't appear to get a one byte event when the UART is first enabled. We just receive the first packet with the first byte missing, and then all subsequent packets are missing the first byte. The length is correct, but the last byte just seems to be whatever was sitting on the buffer. We're able to reproduce this by booting our device on an already busy bus, though it's sporadic and doesn't always happen. It never seems to happen on a quiet bus, but that won't always be the case for us. This is a situation that could happen in our deployment, so a fix would be nice to have. We'll continue looking as well.

RE: Zephyr UART Async receive drops a single character

Hieu — Tue, 20 Feb 2024 09:48:07 GMT

Hi Thilo,

You are right, the alternative I last mentioned requires the Asynchronous API instead of the Interrupt-driven API. Perhaps it would be less risky if you could run a few tests by modifying a minimal sample first rather than with your main project code base?

Cheers,

Hieu

RE: Zephyr UART Async receive drops a single character

C.Thilo — Tue, 20 Feb 2024 07:51:52 GMT

Hi Hieu

Thanks for mentioning it!

But I think this is not a solution for me, as CONFIG_UART_1_NRF_HW_ASYNC depends on UART_1_ASYNC which is not compatible to UART_1_INTERRUPT_DRIVEN ....

But I'm curious, if this works. Perhaps it is worth to refactor all the code to work with async API.

Cheers,
Thilo

RE: Zephyr UART Async receive drops a single character

Hieu — Mon, 19 Feb 2024 22:28:56 GMT

Hi dan_opito and C.Thilo ,

I was recently given some pointers on this topic.

The reason for the miscounted buffer size is because by default RX bytes are counted by handling interrupts on every RX byte. If the interrupt is not handled on time, the counter will miss a byte.

There is an option to enable HW byte counting using TIMER and PPI and then every byte is counted correctly. This is enabled by setting these Kconfig:

CONFIG_UART_1_NRF_HW_ASYNC=y //enabling hw counting
CONFIG_UART_1_NRF_HW_ASYNC_TIMER=2 //picking timer instance used for that

Please let me know if this remedies the issue for you.

Hieu

RE: Zephyr UART Async receive drops a single character

dan_opito — Mon, 30 Oct 2023 19:07:46 GMT

Glad I could help! The external device may be sending something to mcuboot before the application starts - I wonder if another possible workaround would be to reset the peripheral somehow.

RE: Zephyr UART Async receive drops a single character

C.Thilo — Mon, 30 Oct 2023 11:00:18 GMT

Hi Dan

I've exactly the same problem. I can reproduce it when before running our app, I communicate with mcuboot via the same UART. For me it is always a 0x00 byte.
When I do not send anything to mcuboot before, that bug does not occure.

I will try your workaround. Thanks for sharing it!

RE: Zephyr UART Async receive drops a single character

Hieu — Mon, 04 Sep 2023 09:33:02 GMT

Hi Dan,

I am sorry to hear that the issue showed up again.

We have registered it internally, but on DevZone, I cannot comment on the progress or the expected fix date.

If this is critical, then could you please have a word with your local Nordic Regional Sales Manager?

RE: Zephyr UART Async receive drops a single character

dan_opito — Sun, 03 Sep 2023 21:35:30 GMT

I think I've finally seen this issue in the field, even with my fix/hack. we have a hundred or so products in the field that all use this uart method - and I noticed the other day that exactly one of these has stopped working. Upon closer inspection, it is sending the correct number of characters but with an offset of two.

The code in my initial message is still what I am using today - and until I spotted this device (which died two days ago) things have been working well.

Can I get an update on where this is at?

RE: Zephyr UART Async receive drops a single character

dan_opito — Thu, 06 Jul 2023 21:33:23 GMT

No problem, Hieu. I appreciate you taking the time to look into this bug for me. In the meantime, my workaround is working great - we've got this on a couple of different products and I haven't noticed any issues.

RE: Zephyr UART Async receive drops a single character

Hieu — Thu, 06 Jul 2023 09:01:16 GMT

Hi dan_opito,

It has been a while, so I want to give you an update. I have reported the issue internally. However, as developers are away on summer vacations, there has not been any feedback yet.

On that note, due to the same reasons, we are currently understaffed, and further response will be delayed. My apology for the inconvenience.

RE: Zephyr UART Async receive drops a single character

dan_opito — Thu, 22 Jun 2023 03:54:28 GMT

No issue with any delays from our side. The work around is working really well and I haven't noticed any issues. We're not streaming lots of data - but we do have a few devices reporting every 10 minutes that have remained working consistently.

WRT the off-by-one on the buffer size - that's likely left over from me trying to figure out why I was always missing an octet.

Your description above sounds like you may have reproduced the issue. I'm working on another product which will use the same UART and there will be 50 or so of those deployed over the next couple of months - plenty of test vectors and these ones will have higher bandwidth I believe.

If you find a better fix, or if you find the underlying issue, I can push updates out to our devices after a bit of testing in our lab. Thanks for your efforts on this bug!

RE: Zephyr UART Async receive drops a single character

Hieu — Tue, 20 Jun 2023 23:59:07 GMT

P.s: I notice that in handling of UART_RX_BUF_REQUEST, you provide the buffers one byte shorter than their actual length. Is there a particular reason for this?

    case UART_RX_BUF_REQUEST:
    {
        if (!rx_buf1.is_in_use) {
            rx_buf1.is_in_use = true;
            uart_rx_buf_rsp(dev, rx_buf1.buffer, sizeof(rx_buf1.buffer) - 1);
        } else if (!rx_buf2.is_in_use) {
            rx_buf2.is_in_use = true;
            uart_rx_buf_rsp(dev, rx_buf2.buffer, sizeof(rx_buf2.buffer) - 1);
        }

        /* If we didn't manage to hand over a buffer, we'll get the disabled event. */
        break;
    }

RE: Zephyr UART Async receive drops a single character

Hieu — Tue, 20 Jun 2023 23:38:36 GMT

Hi dan_opito,

Sorry for the wait. I ended up getting sick from last weekend and have been only able to work for 1-3 hours each day since.

I was able to create a test setup with my 2x nRF52840 DKs. I produced an issue, however I am not sure if it is the same issue. My issue happens when a buffer is full and the second buffer is switched in. Then I see that the buffer is written correctly, but the offset and length are 1 short in some situations.

This happen regardless of whether the last or first byte transmitted is 0xFF or not.

Interestingly, no byte is lost... However, it does lead to an issue that one byte is forever detected late by one transmission the entire process, until, for again unclear reasons, the state of the buffer becomes correct again.

I will have to discuss this issue to our internal team for analysis. However, as I am on sick leave, that will have to be next week. My apology for this inconvenience.

If the issue I described here is not the same as yours, could you please help me create a minimal setup to reproduce it?

Here is the project I used to observe the issue above. It is based on your code, but I made some renames as I debug it. You could create your issue reproducing project based on it.

In this project, the application logs to UART0 and RTT, and the communication that is tested is over UART1.

devzone.nordicsemi.com/.../c309490_5F00_uart_5F00_char_5F00_230621_5F00_02.zip

Edit: Fix corrupted attachment.

Hieu

RE: Zephyr UART Async receive drops a single character

Hieu — Thu, 15 Jun 2023 16:06:38 GMT

Thank you for the information. I too don't think this issue is exclusive to hardware.

I couldn't get to reproducing it as I had planned, so I don't have anything meaningful to feedback to you yet. My apology. I will continue at it and update you as soon as I find anything.

RE: Zephyr UART Async receive drops a single character

dan_opito — Wed, 14 Jun 2023 21:05:07 GMT

Yes, sorry, this is on our nRF9160 board. We also have an nRF91 Thingy - not sure we have the DK but I think this is likely to be hardware independant. In case not, here's the device tree overlay for it:

&uart2 {
	status = "okay";
	current-speed = < 4800 >;

	pinctrl-0 = <&tunnel_default>;
	pinctrl-1 = <&tunnel_sleep>;
	pinctrl-names = "default", "sleep";
};

&pinctrl {
	tunnel_default: tunnel_default {
		group1 {
			psels = <NRF_PSEL(UART_TX, 0, 15)>,
				    <NRF_PSEL(UART_RX, 0, 13)>;
			bias-pull-up;
			// psels = <NRF_PSEL(UART_TX, 0, 24)>,
			// 	    <NRF_PSEL(UART_RX, 0, 23)>;
		};
	};

	tunnel_sleep: tunnel_sleep {
		group1 {
			psels = <NRF_PSEL(UART_TX, 0, 15)>,
					<NRF_PSEL(UART_RX, 0, 13)>;
			// psels = <NRF_PSEL(UART_TX, 0, 24)>,
			// 	    <NRF_PSEL(UART_RX, 0, 23)>;
			bias-pull-up;
			low-power-enable;
		};
	};
};

You've figured out the RX buffer structure - here's my definition for completeness:

/* Receive buffer structure to manage our data from the device. */
/* NOTE: CONFIG_OPITO_SERIAL_TUNNEL_RECEIVE_BUFFER_LENGTH is set to 256 */

struct rx_buf {
    /* Holds the received data. */
    uint8_t buffer[CONFIG_OPITO_SERIAL_TUNNEL_RECEIVE_BUFFER_LENGTH];
    /* True when the UART driver is using the buffer. */
    bool is_in_use;
};

/* Our ping-pong buffers for receiving data from the device. */
static struct rx_buf rx_buf1;
static struct rx_buf rx_buf2;

The Bluetooth example appears to be disabling once it receives a full packet - but that very well could mask the issue. When reproducing, I'd suggest treating the data as a never-ending stream (rather than packetised transport) - in which case you would never disable the buffer.

It may well be that I've misunderstood how it's supposed to work - but I feel like disabling UART RX is opening ourselves to dropped characters?

RE: Zephyr UART Async receive drops a single character

Hieu — Tue, 13 Jun 2023 18:14:32 GMT

Hi dan_opito,

I cannot see anything obviously standing out in your code just by reading, so I guess I will need to try to reproduce it. Do you observe this issue on a nRF9160?

Meanwhile, the Bluetooth: Peripheral UART sample handles UART event somewhat like your workaround, so I wonder if it is how it is supposed to work.

I don't have a nRF9160 DK ready and also ran out of time today, but I will continue to look into this tomorrow.

By the way, are your RX Buffers something like this?

struct rx_buf
{
	bool is_in_use;
	uint8_t buffer[TEST_BUFFER_SIZE];
};

struct rx_buf rx_buf1 = {0};
struct rx_buf rx_buf2 = {0};

Hieu