nRF52840 Multiple Instance UARTE with HWFC enabled RX bytes occasionally incorrect

[David]

I have a custom board with 3 nRF52840 chipsets, 1 central module is connected to 2 side modules.

The central module communicates with each of the side modules using UARTE with HWFC enabled.

For the most part this is working nicely, however occasionally one of the side modules will receive junk data into its UARTE buffer, even though this was not what was sent.

For example:
Central Module Sent:
13, 00, 03, 02, 01, 20, 02, 02, 00, 0B, 04, 08, 00, 04, 03, 01, 32, 00, 02, 02, 08

Side Module Received:

D3, 17, 49, 20, 7F, BD, A1, B4, DC, A8, 7B, 5A, FB, CD, 2D, 5B, 9B, 9A, C9, 27, DD, C2, 9E, BE, ED, C1, 2A, 8F, 1B, 2D, 62, F2, D9, C7, 57, BF, 36, 06, 13, 2D, 47, D6, 58, 95, 5F, AD, 05, DC, 13, 00, 03, 02, 01, 20, 02, 02, 00, 0B, 04, 08, 00, 04, 03, 01, 32, 00, 02, 02, 08, 65, 61, 64, 65, 72, 0B, 03, 01, D4, 08, 00, 00, D6, 08, 00, 00, 04, 00, 00, 00, F8, 79, 00, 20, A4, 25, 04, 00, 28, 57, 04, 00, 50, 57, 04, 00, FF, FF, 00, 00, 00, 90, D0, 03, 00, 10, 01, 40, 11, 00, 04, 00, FD, 21, 04, 00, 10, 7A, 00, 20, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00

I can see that the message I sent is contained within the bytes received but it should be at the front not somewhere squashed in the middle.

Like I said this only happens occasionally so for the most part this is working, and it usually only happens on 1 of the side modules. The current solution for dealing with this (not really a solution more of a work around). Is to programmatically reset the module if this happens continually for 20 minutes..

I am sending a lot of data quite frequently.

I have 2 UART buffers statically sized at 300 bytes (utilizing the secondary buffer).. So with every UART transmission, 300 bytes of data is sent. the first 2 bytes of my messages dictate the size of the whole message to actually be used.

I am using 1000000 baud rate (I tried to slow down the baud rate and it didn't seem to fix the issue).

Any Help on this would be greatly appreciated

Regards,

David Hutchinson

[Karl Ylvisaker]

Hello David Hutchinson,

UARTE with HWFC enabled.

however occasionally one of the side modules will receive junk data into its UARTE buffer, even though this was not what was sent.

It sounds very strange to me that you are suddenly receiving data unexpectedly. Is the bus between the devices shared with any other functionality? Since you also are using hardware flowcontrol it is highly unlikely that this is 'noise' or similar.

Could the data be unexpectedly queued in the application? Could you check the output buffer of the central module when this happens, and verify that it does not also contain the 'junk' part of the message?

Do you have access to a logical analyzer? It would be greatly helpful if you could scope the lines and see what is happening on the lines when this behavior happens.

Looking forward to resolving this issue together!

Best regards,
Karl

[David]

Karl,

I don't have access to a logical analyzer.

this was sent by the central module (minus the other 279 trailing bytes) when the error occurred.

13, 00, 03, 02, 01, 20, 02, 02, 00, 0B, 04, 08, 00, 04, 03, 01, 32, 00, 02, 02, 08

The issue is very hard to reproduce, it may be running fine for 1 - 2 days and then this happens, Sometimes it happens within 1 - 2 hours.

I have attached the UARTE Driver and the cyclic buffer it uses to store up backlogged messages to be sent.

So how it works is we have a Cyclic Buffer that can hold 7kb of data "Packets". The packets are variable length they are our messages to be sent between the modules. The UARTE driver will pull the next packet out of the cyclic buffer and copy to the UARTE buffer to be sent.

NOTE: All of our messages have a 2 byte length header which indicates how long the rest of the message is.

I will try come up with a way to reproduce this issue more frequently and capture more information CyclicBuffer.h

Uarte.cppUarte.h

Regards,

David Hutchinson

[Karl Ylvisaker]

Hello again, David Hutchinson

David said:

I don't have access to a logical analyzer.

That is unfortunate - it would help immensely in situations like these. I highly recommend taking a look at he options that exist today - some of them are quite cheap, and a logical analyzer will save you a lot of time spent debugging in situations like these.

Thank you for the elaboration on the setup description. Could you confirm whether or not the pins and/or bus is used for anything else than the UART communication between the central and each device?

David said:

The issue is very hard to reproduce, it may be running fine for 1 - 2 days and then this happens, Sometimes it happens within 1 - 2 hours.

David said:

NOTE: All of our messages have a 2 byte length header which indicates how long the rest of the message is.

I see. Could it be an option to save these header bytes for a duration of 1-2 days, and see what they say when/if the error occurs?
From your description I really can not see any way that these bytes might appear on the line, other than that they were sent deliberately (albeit maybe not intentional by the developers of the firmware).

David said:

I will try come up with a way to reproduce this issue more frequently and capture more information

Great! Briefly looking through the code you sent me I unfortunately do not immediately see anything that might cause the behavior you describe.
Are you using both RawWrite and Write? Is it only these two functions in your program that might append things to the TX buffer? Could you for debugging purposes add a check to the functions that will trigger when an unexpectedly (from the developers perspective, like the packet you mention in your original ticket) long packet is attempted written to the buffer?

Looking forward to resolving this issue together!

Best regards,
Karl

[David]

Karl,

I just ran a test to see whether or not the data that is actually getting put onto the UARTE buffer is wrong. and it is not, so somehow over the UART the data is getting scrambled.

Basically my test was pulling the next packet out of Cyclic Buffer then parsing it (the same method used on receiving side). if the parse fails I raise an event. This event was never raised on sending side but was raised on receiving side.

I hope that makes sense. We are looking into getting an Oscilloscope / Logical Analyzer. But in the mean time are there any issues with UARTE that may cause this? 

There shouldnt be anything else using the same lines (unless the softdevice itself assigns pins?).

Pins being used:
SIDE-MODULES:

RTS = 15
RXD = 24
TXD = 16
CTS = 14

MIDDLE-MODULE:

Inst_one:
RTS = 31
RXD = 26
TXD = 6
CTS = 27

inst_two:
RTS = 2
RXD = 16
TXD = 24
CTS = 15

[David]

Karl,

Another thing i would like to mention, I don't know if its related but when the board starts up, i have to delay the initialization of the UART. It seems that if one side initializes before the other side is "powered up" the UART wont work and i need to re-init the uart at a later time.. This is really annoying and problematic having to put in artificial delays.

Is there something i can do by just initializing the UART and it just works when the other side is ready? This causes a sort of weird thing where both sides need to initialize when they know the other side is ready.. but there isn't a way of telling? or is there?

Regards,

David Hutchinson