libUARTE bug: Unexpected RX free input parameter

RE: libUARTE bug: Unexpected RX free input parameter

Krzysztof Chruscinski — Thu, 16 Apr 2020 09:01:26 GMT

[quote userid="16232" url="~/f/nordic-q-a/58986/libuarte-bug-unexpected-rx-free-input-parameter/244639"]reinitializing libuarte on every error does not sound like a good idea.[/quote]

Why?

I know that customers where doing that and there was no issues with that. I know that it has been used in case when complementary SoC was for example in reset. In that case RX line was going down for certain amount of time which lead to continuous error reports. Libuarte was reinitialized on every error and finally when error condition disappeared, libuarte was ready for reception.

I tried to run the test which would trigger such condition but did not capture behavior you described. Also looking at the code, i don't see how it may fail.

Noticed that you are using `NRF_LOG_FLUSH();` in error handler. That operation can be very time consuming (flushing all logs in blocking, synchronous way). I would recommend to remove it.

RE: libUARTE bug: Unexpected RX free input parameter

hmolesworth — Wed, 15 Apr 2020 18:07:52 GMT

Given that you are using nrf logging with potentially colliding invocations, it's worth reviewing the changes made here which has fixed some issues: atomically-protected-against-re-entry-from-interrupt-context

RE: libUARTE bug: Unexpected RX free input parameter

thefool — Wed, 15 Apr 2020 09:13:25 GMT

This is the errorhandler:

        case NRF_LIBUARTE_ASYNC_EVT_ERROR:
        	if (uart_last_error[instance->id] != p_evt->data.errorsrc) {
        		/**
        		 * Look up error in \see nrf52_bitfields.h or search for UART_ERRORSRC_OVERRUN_Msk
        		 */
				debug_error("NRF_LIBUARTE_ASYNC_EVT_ERROR errorSrc x%x", p_evt->data.errorsrc);
				NRF_LOG_FLUSH();
				uart_last_error[instance->id] = p_evt->data.errorsrc;
        	}
        	if (p_evt->data.errorsrc != 0) {
				if (uart_errors[instance->id]++ > LIBUARTE_FAULTS_MAX) {
					core.internal.uart_failed = 1;
				}
        	}
            break;

we do reinit uart after several errors though (when `core.internal.uart_failed == 1`)

However, reinitializing libuarte on every error does not sound like a good idea.

Still, I don't see how the rx_free error should even be possible!?

RE: libUARTE bug: Unexpected RX free input parameter

Krzysztof Chruscinski — Wed, 15 Apr 2020 08:57:50 GMT

how does you error handler look like? I think that the best would be to reinitialize libuarte there.

RE: libUARTE bug: Unexpected RX free input parameter

thefool — Thu, 09 Apr 2020 09:18:15 GMT

OK, thanks! Have a great Easter holiday!

RE: libUARTE bug: Unexpected RX free input parameter

Krzysztof Chruscinski — Thu, 09 Apr 2020 07:01:37 GMT

there might be something wrong when it comes to recovering from the error. I will need to look deeper into that. Will come back to you after Easter.

RE: libUARTE bug: Unexpected RX free input parameter

thefool — Wed, 08 Apr 2020 22:40:16 GMT

Good idea, but no - my fault though: Inconsistent naming!

with "the callbacks to our code", I meant libuarte calling our libuarte_event_handler()
and NOT the callback in our code (ie. libuarte_event_handler() calling uart_rx_complete_callback() )

the rx_free() I pasted the original question is the only place we ever call it.

RE: libUARTE bug: Unexpected RX free input parameter

hmolesworth — Wed, 08 Apr 2020 18:27:12 GMT

Um .. "After all, the callbacks to our code, (where we increment the head of our buffer to mach what DMA already copied, then call the above rx_free() function)"

Doesn't that mean _rx_free is called twice (if called in the callback)? Might be an issue since p_evt-> data and length is not guaranteed to be the latest (updated) values .. (edit: if rx_free not called in callback just ignore this)

    uart_rx_complete_callback(instance, len, data);
    nrf_libuarte_async_rx_free(instance->uart.uart_instance_ptr, p_evt->data.rxtx.p_data, p_evt->data.rxtx.length);

RE: libUARTE bug: Unexpected RX free input parameter

thefool — Wed, 08 Apr 2020 10:57:37 GMT

yes, the issue is not gone. It's gotten harder to get, and I've noticed that it usually comes after some other uart errors.
The case where I most often saw it, was when waking from sleep mode, the nordic wakes the STM, both start UART, and maybe the STM was a bit faster at starting UART, so Nordic would get some errors when uart started. When this was the case, a couple of ms down the line, I'd then get the rx free error. I found a log-file which contains both of these issues:

unleashed_core.c         : 395:  919291: change_conn_params wish 0
unleashed_core.c         :1001:  919291: Waking up: App active
ble_stack.c              : 728:  919291: changed connection parameters to ACTIVE
unleashed_core.c         :1024:  919291: Waking up IMCU...
wr1_config.c             :  93:  919292: Instance 0 speed 1000000
wr1_hardware.c           : 119:  919292: UART DMA Init ins 0 baud 1000000
libUARTE            : (evt) UARTE ERROR Cwr1_hardware.c           :  62:  919292: NRF_LIBUARTE_ASYNC_EVT_ERROR errorSrc xC
libUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR ClibUARTE            : (evt) UARTE ERROR C
unleashed_stm_control_set_ready_to_receive 1
wr1_uart.c               : 199:  919294: Instance 0 uart init baud 1000000
wr1_packets.c            : 385:  919294: packets init
imcu_master.c            :  84:  919294: IMCU master initialized
wr1_packets.c            : 263:  919499: Ins 0: pkt timeout id 00 cmd 0x6F 0x55 0x0A tries left 24
unleashed_stm_control_get_ready_to_receive 0
wr1_uart.c               : 162:  919509: Ins 0: RTR == 0
wr1_frames.c             :  76:  919560: Instance 0 startbyte not first, deleting frame:               000:  80 (1 bytes)
wr1_packets.c            : 248:  919561: Ins 0: Got 1 framing errors
wr1_packets.c            : 263:  919710: Ins 0: pkt timeout id 01 cmd 0x6F 0x55 0x0A tries left 23
unleashed_stm_control_get_ready_to_receive 1
wr1_packets.c            : 209:  919722: Ins 0: Rx retry packet: cmd 0xEF, id 0x00 (retry 3), datalength 2
wr1_packets.c            : 203:  919722: Ins 0: Received an unexpected packet: cmd 0xEF, id 0x 1, datalength 2
wr1_packets.c            : 204:  919723: Packet data: 55 01 
wr1_packets.c            : 203:  919725: Ins 0: Received an unexpected packet: cmd 0xEF, id 0x 2, datalength 2
wr1_packets.c            : 204:  919725: Packet data: 55 02 
wr1_packets.c            : 263:  919925: Ins 0: pkt timeout id 03 cmd 0x6F 0x10 0x9C tries left 24
wr1_packets.c            : 209:  919937: Ins 0: Rx retry packet: cmd 0xEF, id 0x03 (retry 2), datalength 2
wr1_packets.c            : 203:  919939: Ins 0: Received an unexpected packet: cmd 0xEF, id 0x 4, datalength 2
wr1_packets.c            : 204:  919939: Packet data: 10 10 04 
unleashed_model.c        : 390:  919940: IMCU slave enabled
wr1_packets.c            : 186:  919943: Ins 0: Got a request frame available (cmd 0x6F, id 0x 6)
wr1_commands_extended.c  :1054:  919943: Subsystems update: Controls 0 | Settings 0 | GPS 0
wr1_packets.c            : 329:  919944: Ins 0: Will delete reply cmd 0xEF loc 4
wr1_commands_extended.c  : 933:  919946: Camera controls enabled
wr1_packets.c            : 186:  919950: Ins 0: Got a request frame available (cmd 0x6F, id 0x 8)
wr1_packets.c            : 329:  919950: Ins 0: Will delete reply cmd 0xEF loc 6
unleashed_core.c         :1045:  919951: Wake up complete: Going to state IDLE
unleashed_core.c         :1138:  919952: Going to state ACTIVE
wr1_packets.c            : 186:  919954: Ins 0: Got a request frame available (cmd 0x6F, id 0x A)
libUARTE_async      : Unexpected RX free input parameter. length 2 rx_free_cnt 129 rx_buf_size 128

because so much is happening at a very high speed, enabling more debug output usually helps prevent race conditions, and since I was seeing the issue so seldomly, I doubt I'll be able to catch it with more debug output.

what debug output would help? (maybe I can selectively turn on some of it?)

RE: libUARTE bug: Unexpected RX free input parameter

Krzysztof Chruscinski — Wed, 08 Apr 2020 09:06:40 GMT

Do you still have the issue? Maybe you could enable logs in libuarte and provide them, that should give us a hint.

RE: libUARTE bug: Unexpected RX free input parameter

thefool — Thu, 19 Mar 2020 10:42:35 GMT

[quote userid="5227" url="~/f/nordic-q-a/58986/libuarte-bug-unexpected-rx-free-input-parameter/240598"]That's the case for SDK 16.0.0[/quote]

We are using libUARTE from SDK16.0 (copied it to SDK15), if that's what you mean?

OK, so there is protection for the UARTE interrupt, and the other should be protected by priority. So if it's not interrupts interrupting each other, do you have any other idea what could cause this?
As you can see in our code we don't touch `p_evt->data.rxtx.length` in our libuare_evt_handler. We make a copy of it), so I can't see how our code could even cause this issue...

RE: libUARTE bug: Unexpected RX free input parameter

Krzysztof Chruscinski — Thu, 19 Mar 2020 10:07:41 GMT

There are 2 context from which rx data event may come:

- UARTE interrupt on buffer boundary

- TIMER/RTC/app_timer interrupt on timeout detect (no new data)

Libuarte sets UARTE interrupt to higher level than timeout to ensure that UARTE interrupt won't be interrupted by timeout. On the other hand, in `nrf_libuarte_async_timeout_handler` disables UARTE interrupt during its processing to ensure that this context is also not interrupted. That's the case for SDK 16.0.0

RE: libUARTE bug: Unexpected RX free input parameter

thefool — Mon, 16 Mar 2020 11:47:39 GMT

Just checked: which interrupts are handled and which priority they're given is all done in the driver! Or does the SDK disable all interrupts before calling a handler?
even if the sdk disables the interrupt that it is currently handling, my worry is that a second interrupt from a different source is interrupting an interrupt, but they both call the same callback functions. I don't see any safety measures for this case in the driver code.

RE: libUARTE bug: Unexpected RX free input parameter

thefool — Mon, 16 Mar 2020 10:50:32 GMT

how would the sdk handle such race conditions? doesn't that need to be handled in the driver itself?

RE: libUARTE bug: Unexpected RX free input parameter

Krzysztof Chruscinski — Mon, 16 Mar 2020 10:42:57 GMT

i'm asking because race conditions like that were fixed in SDK 16.0. Could you ensure that you are on SDK 16.0 version? Moving to 16.0.0 should require porting as file location and API changed.

RE: libUARTE bug: Unexpected RX free input parameter

thefool — Mon, 16 Mar 2020 10:33:53 GMT

We are currently using 15.0, but have also tested 15.3 and 16.0. we copied libUARTE from SDK 16.0

RE: libUARTE bug: Unexpected RX free input parameter

Krzysztof Chruscinski — Mon, 16 Mar 2020 10:07:19 GMT

Hi,

which SDK version is that?

RE: libUARTE bug: Unexpected RX free input parameter

thefool — Fri, 13 Mar 2020 13:46:42 GMT

Hi Edvin,

unforutnately no. This happens seldomly in our main project, and we have not been able to narrow it down to a specific case yet.

Could you check if the implementation is safe for race conditions with the two interrupts possibly interrupting each other?

RE: libUARTE bug: Unexpected RX free input parameter

Edvin — Fri, 13 Mar 2020 11:32:01 GMT

Hello,

Is there any way for me to recreate this issue?

Can't really say that I know the cause/fix for this. Is there any way that I can recreate the issue in the office? Is it possible to recreate with one or two DKs?