UART RX using the AsyncAPI generates massive IRQ load

RE: UART RX using the AsyncAPI generates massive IRQ load

ovrebekk — Thu, 16 Nov 2023 13:09:16 GMT

Hi Jürgen

Thanks for the detailed report.

It is interesting that the shorter timeouts would affect the buffering. I assume the number of RX ready events is more or less the same since there is no gap between bytes, but obviously there will be more interference from the timer interrupts.

[quote user="McVertex"]Does this point to a possibility to replace the default software timers by hardware timers ?[/quote]

Is the UART_0_NRF_HW_ASYNC configuration enabled or not in your case?

In ASYNC mode it is necessary to count the number of bytes received since you might want to forward the data to the application before the buffers fill up (timeout), but the underlying UARTE peripheral doesn't make this count easily available. For this reason there are two available methods to handle the byte counting.

There is a software method which relies on an interrupt every time a byte is received, or a hardware method that uses a PPI channel and a TIMER module to count the bytes in hardware.

The latter method is more efficient and reliable, but requires the use of an additional TIMER module which you might need for other purposes.

So enabling this configuration will not solve the issue of getting a lot interrupts when you use a low timeout value, but if you have been testing with it disabled so far it could be interesting to redo your tests with this configuration issue enabled to see if it makes the buffering more reliable when using low timeout values.

Best regards
Torbjørn

RE: UART RX using the AsyncAPI generates massive IRQ load

McVertex — Wed, 15 Nov 2023 13:48:34 GMT

Hi Torbjørn,

maybe not dropped bytes, it seems to be more subtle - seems that "something is wrong" with the read position of the used zephyr ring buffer, see in the following (sorry for the length) description:

In our application we have the following scenario:

a host application sends requests as packets via uart to the nRF5340
the packet length varies from request to request ~10 ... 240 bytes
the baud rate is 115200 baud
there are no gaps between the bytes inside the packet
the nRF5340 receives the packets via uart using the async api
the size of the dma buffers (RX_BUF_REQUEST) is 120 bytes
the packet receiving part is aware of packets being splitted into multiple
rx events, since this happens regularly when there is a switchover to the next dma buffer
All received parts of a packet are copied into a zephyr ring buffer and via a message queue, the packet receiving part in the main thread is signaled
the packet receiving part reads from that ring buffer and checks the completeness and consistency of a received packet by CRC
if the packet is complete and consistent, it is processed by the application
after processing the request packet, a reply packet sent back to the host

Using a rx timeout value in the range below 500us the packet consistency check fails in roughly 1% of the packets.

Using a rx timeout value of 500 us the consistency check fails less often (about 0.1%).

The analysis of received data in case of a failed consistency check showed that the content of the ring buffer contains all received data, but the readout position of the ring buffer seemed to be off by 2 bytes. Once happened, this offset error remains, even if new packets arrive, which led to one missing response and following responses that are out of sync with the requests...

We searched for errors in the code, but did not find any reason that could cause this strange behavior. Thereupon we analyzed the application with SystemView and were surprised by the high IRQ load...

Having no special requirement for the response time, we increased the uart rx timeout to 16 ms. The only reason to choose 16 ms was to reduce the ISR load below 1%, see the measurements in my second post.

Using 16ms for the uart rx timeout, we never observed failed consistency checks...

So currently i think a safe level for rx timeout highly depends on the application and maybe especially on the already existing IRQ load without uart.

IMHO a good advice would be to use a rx timeout value "as high as possible" and to point out that a (unexpected) IRQ load is generated that may cause problems. Especially timeout values below ~5ms, since this will lead to a IRQ frequency of 1kHz and above.

In general it would be better to have the option to use a hardware timer for that purpose.

Searching for informations about this i found the following:

config UART_0_NRF_HW_ASYNC
	bool "Use hardware RX byte counting"
	depends on UART_0_NRF_UARTE
	depends on UART_ASYNC_API
	help
	  If default driver uses interrupts to count incoming bytes, it is possible
	  that with higher speeds and/or high cpu load some data can be lost.
	  It is recommended to use hardware byte counting in such scenarios.
	  Hardware RX byte counting requires timer instance and one PPI channel

config UART_0_NRF_ASYNC_LOW_POWER
	bool "Low power mode"
	depends on UART_0_NRF_UARTE
	depends on UART_ASYNC_API
	help
	  When enabled, UARTE is enabled before each TX or RX usage and disabled
	  when not used. Disabling UARTE while in idle allows to achieve lowest
	  power consumption. It is only feasible if receiver is not always on.

config UART_0_NRF_HW_ASYNC_TIMER
	int "Timer instance"
	depends on UART_0_NRF_HW_ASYNC

Does this point to a possibility to replace the default software timers by hardware timers ?

It even contains the statement "higher speeds and/or high cpu load some data can be lost"...

Unfortunately, i have not yet found sufficient documentation on these configuration values, or did i miss something ?

Best regards,

Jürgen

RE: UART RX using the AsyncAPI generates massive IRQ load

ovrebekk — Tue, 14 Nov 2023 13:42:05 GMT

Hi Jürgen

You mean to say you were dropping bytes when using lower timeout values?

If so we should probably try to figure out the range of values that will work reliably, and update the driver and documentation to warn the user if they try to use a value that is too low.

If you can provide a bit more information about the issues you experienced (other than high CPU usage) I will discuss it with the developers, and request an update either in the docs or the driver.

Best regards
Torbjørn

RE: UART RX using the AsyncAPI generates massive IRQ load

McVertex — Mon, 13 Nov 2023 16:43:11 GMT

Hi Torbjørn,

Thanks for the feedback.

I think it would be great to point this out to the users of the async api, as long as the
implementation shows this behavior since the cause of possible errors is not obvious.

In our case, we had sporadic communication errors in our application - we did not
expect such a relation to the timeout duration 😉

Best regards
Jürgen

RE: UART RX using the AsyncAPI generates massive IRQ load

ovrebekk — Mon, 13 Nov 2023 14:47:59 GMT

Hi Jürgen

I discussed this with the developers, and they provided some details of the driver implementation which seems to explain your findings.

To avoid using too many hardware timer modules the timeout timer is implemented using a software timer, and instead of resetting the timer every time a byte is received the timeout timer is running continuously, at an interval equal to the timeout value divided by 5.

When the interrupt has run 5 times consecutively without any RX bytes received the RX timeout will be triggered.

So when you have a timeout duration of 500us you would expect a new interrupt every 100us, which seems to more or less match your original diagram.

Knowing this I agree with you that the implementation is not very efficient, and using small timeout values is not recommended for the time being. In particular it should be relatively easy to change the driver to stop the timeout timer when no data is received, and I have requested if this can be improved in a future update.

Best regards
Torbjørn

RE: UART RX using the AsyncAPI generates massive IRQ load

McVertex — Fri, 10 Nov 2023 13:41:45 GMT

Hi Torbjørn,

Thank you for the response, i already suspected that, but the current behavior is hardly acceptable
since i (and possibly others too) expect the following:

no CPU load while no data is coming in
no CPU load during data reception of a data packet (as long there is no switchover to another DMA buffer)
a single ISR call when some data came in and the given timeout has elapsed

I expect this behavior because many different STM32 MCU's use exactly this pattern for DMA reception.

I'm quite new to the nRF MCU's and Zephyr, so i'm not deep inside the details of the peripheral units and drivers...

But i never expected a heavy ISR load as seen here, using a relatively moderate timeout duration of 500µs...

In our application we can workaround this by increasing the timeout to 16 ms, since we do not have a hard
requirement on this.

BTW, i already checked the rx data having no gaps:

Now this is what a complete request and reply packet with a 16 ms timeout look like (115200Baud):

If there is any way to eliminate the ISR load produced by the async driver implementation i would highly recommend to do so since this is promised by the async api and IMHO expected by most users.

Best regards,

Jürgen

RE: UART RX using the AsyncAPI generates massive IRQ load

ovrebekk — Fri, 10 Nov 2023 11:00:34 GMT

This sounds like expected behavior the way the underlying UARTE peripheral works with the async driver.

When you have such a short timeout the driver might generate a timeout for every byte transmitted, unless the bytes are sent completely back to back.

For optimal performance with UART you want to let the DMA buffers fill up before having to process the data, and the larger the buffers the less CPU impact.

Is fast response to incoming data a requirement, since you are testing such short timeout values?

In order to analyze this further it would be interesting to see the data stream on a scope, to see whether or not data is sent back to back, or if there are a lot of gaps in the data.

Best regards
Torbjørn

RE: UART RX using the AsyncAPI generates massive IRQ load

McVertex — Fri, 10 Nov 2023 08:12:48 GMT

Additional informations:

The IRQ overhead directly depends on the timout value passed to uart_rx_enable():

timout [us]	IRQ overhead [%]	ISR 37 period [ms]
100	23	0.122
200	23	0.122
500	21	0.122 ... 0.152
1000	14	0.213
2000	7.1	0.427
4000	3.7	0.824
8000	1.9	1.6
16000	0.95	3.2
32000	0.48	6.4

A timeout of 16 ms is necessary to reduce the IRQ overhead to less than 1%.

Is there a way to reduce that overhead ?

Maybe it's unintended behaviour of the async rx implementation ?

All measurements done with
- nRF SDK 2.5.0
- hardware: nRF5340-DK
- build configuration: nrf5340dk_nrf5340_cpuapp
- build optimization level: default
- application: fund_less5_exer1_solution (see previous post)
- extended prj.conf to enable tracing (see previous post)
- only RECEIVE_TIMEOUT has been changed between measurements

Regards