nRF91 modem watchdog fault when re-activating LTE

[JordanYates]

I have an application with a shipping mode that when enabled/disabled calls `conn_mgr_all_if_connect/conn_mgr_all_if_disconnect`.
These functions eventually end up in `lte_net_if.c`, running `lte_lc_func_mode_set` with either `LTE_LC_FUNC_MODE_ACTIVATE_LTE` or `LTE_LC_FUNC_MODE_DEACTIVATE_LTE`.

Connecting to the LTE network on boot works fine, as does entering shipping mode.
The modem fault occurs when attempting to re-activate the modem after it has already been active.
If less than a minute has passed since the modem was de-activated, it re-activates fine and the application continues as per normal.
If over a minute has passed, the modem faults more or less immediately with:

[00:01:26.754,638] <inf> app: Device activated, enable LTE
[00:01:26.982,513] <err> nrf_modem: Modem has crashed, reason 0x2, PC: 0x468de

This issue is 100% reproducible (happens every time), always with the same error code and program counter.
The error code corresponds with NRF_MODEM_FAULT_HW_WD_RESET.

Replacing `LTE_LC_FUNC_MODE_ACTIVATE_LTE` and `LTE_LC_FUNC_MODE_DEACTIVATE_LTE` with `LTE_LC_FUNC_MODE_NORMAL` and `LTE_LC_FUNC_MODE_OFFLINE` does not change the behavior of the fault.

nrfxlib version: Tag v3.1.1 (Commit 3b210a24d3bc7ecfc268e0feab6436306b11e7cb)
Modem firmware: mfw_nrf91x1_2.0.4
Modem model: nRF9151-LACA

[Syed Maysum Abbas Zaidi]

Hi,

We have reviewed this internally, but unfortunately we will not be able to prioritize further investigation of this issue at this time.

Since there is no planned follow-up from our side, I will close this case now.

If the issue becomes critical for your project or if you have new information that changes the situation, please open a new case and reference this ticket.

Best Regards,
Syed Maysum

[AndersHedberg]

Hi, I also hit this problem. But RTT is disabled, and logs are using the UART. 

I'm running on a nrf9151 (Makerdiary dev board), with the same mfw_nrf91x1_2.0.4 version. 

My AI claims that if there is a wait of ~70 seconds or longer after the shutdown this trigger the watchdog error in the modem. 
Question - Is it safe to just retry ...init() and continue if we get no error?

My thinking is that the modem core watchdog was not stopped on shutdown, it might not be possible once it started? But this do consumes energy if it keeps resetting that core? If a reset disables the watchdog, can't the stopped core simply be reset when stopped to avoid reporting this error?

This was found when Codex running 5.6 Sol Ultra set up our CI tests, some tests check the LTE connectivity. 

Unfortunately, when I tell the AI to write a good bugreport, it triggers the security checks in the LLM and just shows me "This content can't be shown... cybersecutiry...  apply for Trusted Access". 

(Below are a preliminary result from the AI investigation, that I managed to capture... )
----

Our strongest retained sequence is:

1. The preceding application completed `CFUN=0`, the NCS shutdown wrapper returned zero, the exact
intercepted `nrf_modem_shutdown()` call returned zero, and `nrf_modem_is_initialized()` was
false.
2. The modem remained dormant for at least 49m32s. During that interval the application core went
through our protected reset/flash path; no RTT reader or modem trace path was used. The pyOCD
reset process exited before the host issued `START` and the initialization call; CMSIS-DAP stayed
physically connected, and `C_DEBUGEN` was not sampled at the exact fault instant.
3. On the next application boot, the first `nrf_modem_lib_init()` returned `-5`. The application
fault callback recorded `NRF_MODEM_FAULT_HW_WD_RESET` reason `0x002`, PC `0x000468de`, no DFU
callback, and false library initialized state.
4. One immediate second `nrf_modem_lib_init()` in the same CPU boot, with no intervening reset,
returned zero. The modem then reported firmware 2.0.4, the UICC was ready, LTE-M registered in
2,818 ms, and checked CFUN0/direct shutdown completed successfully.

We observed the same `-5` / reason `0x002` / PC `0x000468de` initialization signature in two
independent boots. Our app uses NCS 3.3.1, nrfxlib checkout
`632dc0afa66add5d1755058da47437515a0b2f55`, embedded modem-library build
`3.3.1-cellular-88400a57b51c`, and app-side static `libmodem.a` SHA-256
`d99ce88b8f4ccbbdf7b139c7db145e04208673267868dfd753d852ae95fd239b`. We can supply
privacy-redacted serial excerpts privately.