nrf_modem_lib_shutdown() before rebooting into MCUboot serial recovery: modem "Reset loop" event and SecureFault inside the precompiled Modem library (handle_modem_rpc_msg) on the next application boot

[RogerLee]

Environment

- nRF Connect SDK v3.4.1 (nrfxlib v3.4.1), modem firmware 2.0.4

- Board: nRF9151 DK, target nrf9151dk/nrf9151/ns, sysbuild with MCUboot serial recovery

- Application: LTE-M/NB-IoT session with lte_lc + MQTT/TLS (modem active before the DFU entry)

Flow and observation (2 out of 3 attempts)

1. DFU entry from the application: lte_lc_offline(); nrf_modem_lib_shutdown(); sys_reboot(SYS_REBOOT_COLD);

2. MCUboot serial recovery: upload a new application image (nrfutil mcu-manager), then `serial reset`.

3. On the next application boot we observed:

   - log: `Modem domain event: Reset loop` (LTE_LC_MODEM_EVT_RESET_LOOP), and

   - a non-secure SecureFault taken from the modem IPC interrupt path:

         Exception came from non-secure FW in handler mode.

         LR: 0x000411f9  PC: 0x00041194   R6: 0x00000000

         SFSR: 0x00000048 (AUVIOL|SFARVALID)  SFAR: 0x00000000

   Disassembly shows PC=0x41194 inside handle_modem_rpc_msg in the precompiled libnrf_modem (`ldr r3,[r6]` with r6 = NULL - an RPC queue pointer that is not initialized yet), reached from rpc_message_handler via the modem IPC interrupt.

Counter test (no shutdown)

If the application just calls sys_reboot() without shutting the modem down, the same dfu -> recovery -> `serial reset` flow was clean: 6 out of 6 consecutive loops, no Reset loop event, no fault. An additional isolation test (set the recovery boot mode from a debugger, cold reboot into recovery, no modem interaction) was also clean.

Questions

1. Is calling nrf_modem_lib_shutdown() right before rebooting into MCUboot serial recovery (application-only DFU) a supported sequence? The Modem bootloader documentation requires nrf_modem_shutdown() only for a modem firmware update; our flow updates only the application.

2. Is the handle_modem_rpc_msg NULL-queue crash a known issue when the modem is re-initialized shortly after a graceful shutdown plus two resets? Note: per the %MDMEV documentation the Reset loop restriction counts more than 7 resets without a graceful +CFUN=0, but here the modem was shut down gracefully before the first reset and only two resets happened - so we do not understand how the event was triggered either.

3. What is the recommended sequence for entering serial recovery from a running application: keep the modem running, or shut it down first (and in that case, anything else to observe)?

Current workaround: we removed the shutdown from the DFU entry - the modem stays running across the reset into recovery and the failure no longer occurs. No negative effect observed so far. We can provide full register dumps, the application ELF and further logs on request.

[Hakon]

Hello,

as a further test, can you try adding a k_msleep(500)) after nrf_modem_lib_shutdown()? Also, it would be really helpful if you could provide a modem trace.

[RogerLee]

Hi Hakon,

thanks for the follow-up - here are the results of the k_msleep(500) test, with modem trace capture attempted across the whole cycle.

Setup used for every run below

- nRF Connect SDK v3.4.1, modem firmware 2.0.4, nRF9151 DK (nrf9151dk/nrf9151/ns, sysbuild with MCUboot serial recovery).

- Trace build: official nrf91-modem-trace-uart snippet applied to the application image only (-Dnrf9151_SNIPPET=...), trace level FULL (the Kconfig default). Trace captured over UART1 (second VCOM, 1 Mbaud) with a small local raw reader (pyserial) - the nrfutil trace on this machine could not finish its trace-database install.

- Each run: enter recovery from the application (fmg dfu - see variants below) -> MCUboot serial recovery -> nrfutil mcu-manager serial image-upload -> serial reset -> next application boot.

- The capture was started before the shutdown and left running past the next application boot, so each window covers a full cycle on the host side (see the caveat below about what the stream actually contains).

Results

- variant "shutdown + k_msleep(500)" (fmg dfu shutdown, 500 ms pause), **6 runs: 5 clean boots, 1 failed with the same fault** (details below).

- variant "shutdown without pause" (fmg dfu shutdown 0, the original failing sequence), 3 valid runs: all boots clean. (One further run was aborted before the upload due to a host-tool timeout and is excluded.)

- reference "modem left running" (default fmg dfu), 1 run: clean.

Failure with the 500 ms pause in place (run "B6")

- Same signature as our original report - non-secure SecureFault, handler mode, SFSR 0x48 (AUVIOL|SFARVALID), SFAR 0x0, R6 0:

```

*** Booting nRF Connect SDK v3.4.1-b20f8619ba9a ***

Reset reason: software

SecureFault

    Exception came from non-secure FW in handler mode.

    MSP_NS:  0x20033000   PSP_NS:  0x20033f88

        R3:   0x00041f11   LR:   0x00041ef9   PC:   0x00041e94

        R6:   0x00000000

    SFSR:  0x00000048

    SFAR: 0x00000000

```

- PC/LR are 0x41e94/0x41ef9 in this build; in the build of the original report they were 0x41194/0x411f9 (+0x300 shift for the same code region in the precompiled modem library). The fault happens extremely early (right after the Zephyr banner, before any application log line), consistent with the earlier observation, and the device remains stopped afterwards.

Traces

- Files (raw modem trace streams over UART1, second VCOM, 1 Mbaud; open them in Cellular Monitor with modem trace database mfw_nrf91x1_2.0.4, or convert to PCAP in Wireshark):

  - `trace-B6-shutdown500.raw` (1.16 MB) - the failing run's window: **no trace bytes arrived from the shutdown sequence through the failing boot**; after a manual reset the recovered device streamed 1.16 MB (boot + modem initialization + running application). The fault itself is evidenced by the console dump above.

  - Clean-cycle windows: `trace-C3retry-shutdown0.raw` (885 KB), `trace-C3-shutdown0.raw` (642 KB), `trace-B1-shutdown500.raw` (257 KB), `trace-B5-shutdown500.raw` (129 KB), `trace-smoke-test2.raw` (890 KB).

- Caveat about our capture setup: the raw stream arrives in bursts - hundreds of KB around a boot / modem (re-)initialization, and quiet in between. In the failing run's window the stream produced no bytes at all until the next reset, and the failing boot itself (crash within the first few hundred ms) produced no trace output before it faulted. We are not sure yet whether the shutdown phase can be captured at all by tracing this way.

- If you want a specific window captured with a specific tool (e.g. nRF Connect Cellular Monitor live capture, or nrfutil trace with its database installed) or with different trace settings, please say so - the failure is timing-dependent, so reproducing it again may take a number of attempts.

- Since these traces contain device/network identifiers (IMEI, cell IDs, IP addresses, operator and server names), we would prefer to share them through a private channel rather than attaching them publicly. If a support case is the right place for them, we will attach them there - please advise what works best on your side.

Observations

- The 500 ms pause does not reliably prevent the failure - it occurred with the pause in place (1 of 6 runs).

- Reproduced rate in this session: 1 failure in 10 full shutdown cycles (lower than the 2 of 3 we initially saw; the failure seems timing-dependent).

- Separate observation (may or may not be related): calling the shutdown sequence very early after boot (about 10 s after a reset, while the modem is still searching) hung once in lte_lc_offline() without returning; steady-state calls (about one minute after boot) never hung. We can provide details if useful.

- Happy to run more repetitions, longer pauses, or capture a specific window if that helps.

Best regards,

RogerLee

[Hakon]

I don't see any of the modem traces in the attachment. Can you upload the files to this ticket?

[RogerLee]

Since these traces contain device/network identifiers (IMEI, cell IDs, IP addresses, operator and server names), we would prefer to share them through a private channel rather than attaching them publicly. If a support case is the right place for them, we will attach them there - please advise what works best on your side.