Problems using modem_cellular driver with the modem_backend_uart_isr option in ncs-3.4.0

[Nathan Boyd]

Hi,

We are developing a power sensitive application using the nrf54LM20A with a Quectel EG916Q modem (with our own provided out-of-tree driver handing the EG916Q scripts).

We have noticed a problem with using the nordic UARTE driver for zephyr via the zephyr ISR API connected to the ISR version of the modem_backed_uart driver, with runtime device power management enabled.

All functions well except that under many different situations, the modem_cellular driver closes modem_backend_uart_isr with the PM usage count imbalanced, stopping the UARTE peripheral from being automatically suspended at the end of usage.

Looking at the code, I notice a few things that seem incompatible between the uart_isr modem backend and the nordic UARTE driver:

• The modem_backend appears use uart_irq_tx_disable() and uart_irq_tx_enable() around putting tx data into the tx ring buffer in modem_backend_uart_isr_transmit(), rather than using a mutex or some other specific lock. This in itself seems OK, as the calls are matched.
• The zephyr ISR UART API for the UARTE driver seems to use the PM calls (pm_device_runtime_get() and pm_device_runtime_put_async()) to "hold" the UARTE device during some operations (such as, for example, disabling tx, as this happens somewhat asynchronously)
  • If uart_irq_tx_enable() is called too soon after uart_irq_tx_disable(), before the disable actually finished, uart_irq_tx_enable() "gets" the pm runtime usage without the previously expected "put" being done.

The usage pattern I saw with the modem_cellular driver, talking to the EG916Q at 921600bps meant that almost every AT command resulted in the usage count creeping up.

My questions are:

• Is the modem_backend_uart_isr code using uart_irq_tx_disable/enable() expected and valid? Or is it likely to change.
• Is there any workaround or fix for the UARTE driver in progress?

We have also tried using the modem_backed_uart_async option, which balances the PM usage correctly, but suffers from the known RX timeout problem with UARTE. We are currently trying the async driver with the CONFIG_UARTE_NRFX_UARTE_COUNT_BYTES_WITH_TIMER Kconfig option, which seems use a separate timer peripheral for the RX timeout, but we do see some performance problems (some CMUX frames missing bytes), so it isn't an easy win.

I am currently considering changing the modem_backend_uart_isr code to protect the TX ringbuffer another way, but I would rather stick to known unmodified nRF Connect zephyr code.

Has anyone else experienced similar problems?

Regards,

Nathan Boyd.

[Vidar Berg]

Hello Nathan,

I will forward this to the developers to check if they can confirm the issue with the ISR version. Will update the ticket here as soon as I have heard back from them. Regarding missed frames with the async driver, maybe you did it already, but enabling HW byte counting on nRF54L series with the TIMER/COUNTER requires specifying the timer instance to be used under the uart devictree node like done here: https://github.com/nrfconnect/sdk-zephyr/blob/56e06d37288a9b25d0f399faefbda439134d1dd2/tests/drivers/uart/uart_async_api/boards/nrf54l15dk_nrf54l15_cpuapp.overlay#L45. Documentation on how to enable reliable reception with HWFC for reference: https://nrfconnectdocs.nordicsemi.com/ncs/latest/nrf/drivers/peripheral_drivers/uart.html#reliable-reception-on-nrf54l-and-nrf54h-series. Also, probably unrelated to the missed data, but I recommend enabling CONFIG_UART_NRFX_UARTE_HFXO_ON_ACTIVE if you aren't already requesting the HF crystal oscillator from your application. The HFXO ensure that you will have a sufficiently accurate baud rate across temperature and devices.

Regards,

Vidar 

[Nathan Boyd]

Hi Vidar,

We aren't currently using CONFIG_UART_NRFX_UARTE_HFXO_ON_ACTIVE, but we have previously confirmed that we always have the HFXO running when we need this UART.

We have tested with the separate TIMER/COUNTER specified to handle the RX timeout, and this works in terms of not ending up with the RX disabled randomly, but the CPU load seems so high when running at 921600bps that the CMUX driver is dropping bytes. I am not sure if this is dropping at the modem backend level, or at the CMUX driver (on ncs-3.3.0 with earlier modem code we had a lot of trouble tuning all the buffer sizes through the modem and network stack to make sure that we were not dropping data there.

Our application is rather simple from a power management point of view, as we spend most of our time idle, but using PM_DEVICE_RUNTIME simplifies a few of our use cases, so we do rely on the UART being suspended when not in use.

We will try the CONFIG_UART_NRFX_UARTE_HFXO_ON_ACTIVE to see if there is a chance it is related to clock precision, as I did suspect in an earlier ticket (case ID 344534) relating to async UART problems that frame errors were detected, leading to the RX being aborted.

Regards,

Nathan.