I was running into issues with issuing the AT#XRESET command for a PPP/CMUX configuration. In an attempt to make the setup as robust as possible we found that the 9151 stopped responding to AT commands. This is on the latest serial modem with board specific changes rebased on top. Here's the full breakdown:
ENVIRONMENT
- nRF9151 Feather, modem firmware mfw_nrf91x1_2.0.4
- NCS v3.5.0-preview3, sdk-zephyr ae4cf2e3a22 (Zephyr 4.4.99)
- drivers/serial/uart_nrfx_uarte.c, async API on uart0, 115200 baud, no HW flow control
- CONFIG_UART_0_NRF_HW_ASYNC=y, CONFIG_UART_0_NRF_HW_ASYNC_TIMER=0
(CONFIG_UARTE_NRFX_UARTE_COUNT_BYTES_WITH_TIMER_LEGACY=y), CONFIG_PM_DEVICE=y
- Application: Serial Modem (ncs-serial-modem), AT host on uart0
SYMPTOM
After a software reset (AT#XRESET, i.e. sys_reboot) while the host keeps transmitting, the application boots normally but never receives any data on the UART again, until a pin reset. The application callback gets no UART_RX_RDY, no UART_RX_STOPPED and no UART_RX_DISABLED, so it cannot detect or recover from the condition.
REPRODUCTION
1. Serial Modem idle in AT command mode.
2. Host sends "AT#XRESET\r", then immediately streams data at 115200 for about 3 seconds (CMUX close-down frames plus "AT\r" every 5 ms), so the line stays busy while the nRF reboots.
3. After the boot, the host sends "AT\r".
Without the fix below: no answer (only 1 of 5 attempts recovered).
With the fix: 10 of 10 attempts answered within 0.55 s.
STATE IN THE STUCK CONDITION (core halted with GDB)
UARTE0 registers:
- ENABLE = 8, PSEL.RXD = P0.10, PSEL.TXD = P0.11, ERRORSRC = 0
- RXD.PTR = the driver's RX buffer, RXD.MAXCNT = 2048
- EVENTS_RXDRDY = 1, EVENTS_RXSTARTED = 0, EVENTS_ENDRX = 0, EVENTS_RXTO = 0
- INTEN = 0xA0210 (ENDRX, ERROR, RXTO, RXSTARTED)
- The RX DMA buffer holds everything the host sent after the boot, including every later "AT\r". The hardware is receiving.
Driver state (uarte_0_data.async->rx):
- enabled = false, stopped = true, discard_fifo = false
- total_user_byte_cnt = 0, RX k_timer not running
Ruled out:
- Byte counting: UARTE0 PUBLISH_RXDRDY and TIMER0 SUBSCRIBE_COUNT are both on DPPI channel 15, and the channel is enabled.
- Pins: RX is an input with its buffer connected, idling high. TX is an output, idling high.
- SPU: GPIO and UARTE0 are non-secure, as expected.
ANALYSIS
1. While RX is disabled during boot, the incoming stream latches line errors (framing/break), so EVENTS_ERROR is set.
2. The application calls uart_err_check() before uart_rx_enable(). That clears ERRORSRC but not EVENTS_ERROR.
3. uarte_nrfx_rx_enable() enables the ERROR interrupt and triggers STARTRX without clearing the stale event, so the ISR runs immediately.
4. error_isr() calls rx_disable(dev, false), which sets enabled = false and stopped = true and triggers STOPRX before RX has actually started (RXSTARTED not seen yet).
5. That STOPRX appears to be lost: RXTO is never generated and the receiver keeps running. The driver waits for an RXTO that never comes, so rx_disable_finalize() (UART_RX_DISABLED) is never called and received data is never reported.
Step 5 is inferred from the register state above. We have not measured the STARTRX to STOPRX timing.
FIX TESTED
In uarte_nrfx_rx_enable(), before interrupts are enabled and STARTRX is triggered:
--- a/drivers/serial/uart_nrfx_uarte.c
+++ b/drivers/serial/uart_nrfx_uarte.c
@@ -1924,6 +1924,14 @@ static int uarte_nrfx_rx_enable(const struct device *dev, uint8_t *buf,
async_rx->next_buf_len = 0;
async_rx->stoprx_deferred = false;
+ /* Errors latched while RX was disabled (e.g. the peer transmitting across a
+ * reset) do not belong to this reception. Left pending, the ERROR interrupt
+ * fires right after STARTRX and its STOPRX can be lost before RX has started,
+ * leaving RX running with the driver considering it stopped.
+ */
+ nrf_uarte_event_clear(uarte, NRF_UARTE_EVENT_ERROR);
+ (void)nrf_uarte_errorsrc_get_and_clear(uarte);
+
if (IS_ENABLED(CONFIG_PM_DEVICE_RUNTIME)) {
pm_device_runtime_get(dev);
} else if (LOW_POWER_ENABLED(cfg)) {Result: 10 of 10 reset-under-traffic cycles recovered, against 1 of 5 before.
This fix is on the vanilla Zephyr side. Happy to head over there to drop a PR to address this if you guys thing it's worth it.
