nRF9160 on a custom board having modem crash, reason 0x10

We have this custom sensor board, running nRF9160.

Application code built with SDK 2.5.0, running modem FW 1.3.2, and using LTE-M connecting to MS Azure IoTHub with MQTT.

Mostly, our asset tracker application runs just fine, but in rare cases the modem calls starts to fail, and the SDK error log reports that the modem has crashed.

This is (the essence of) our error log when the error occurs:

[34:31:47.742,065] <err> nrf_modem: Modem has crashed, reason 0x10, PC: 0x119f0a
[34:33:17.807,312] <err> azure_iot_hub: mqtt_helper_connect failed, error: 11
[34:33:17.807,342] <err> cloud_handler: Failed to connect to Azure IoT Hub, error 11
[34:33:17.917,999] <err> lte_lc: Failed to set functional mode. Please check XSYSTEMMODE.
[34:33:17.918,029] <err> cloud_handler: Failed to disconnect from the network (Connect mode 2, PSM mode 0), error -14
[34:33:20.918,151] <err> lte_lc: Failed to set functional mode. Please check XSYSTEMMODE.
[34:33:20.918,182] <err> cloud_handler: Failed to disconnect from the network (Connect mode 2, PSM mode 0), error -14
[34:33:23.918,304] <err> lte_lc: Failed to set functional mode. Please check XSYSTEMMODE.
[34:33:23.918,334] <err> cloud_handler: Failed to disconnect from the network (Connect mode 2, PSM mode 0), error -14
[34:33:26.918,579] <err> lte_lc: Failed to set functional mode. Please check XSYSTEMMODE.
[34:33:26.918,609] <err> cloud_handler: Failed to disconnect from the network (Connect mode 2, PSM mode 0), error -14

This might happen after a few days of running, after several weeks, or just (mostly) never happening.

As you might see from the logs the first error=11  is returned when trying to connect from the call to lte_lc_connect_async().

Of course there are a lot of connect calls performed earlier that works just fine, but here the modem seems to have crashed and causing the error.

  1. Is is possible to explain anything regarding the crash reason (being 0x10) reported in the error message?
  2. Is is possible to recover the modem after such a crash, or is a reset the only way?
  3. What is the state of the modem (or SIP) after such a crash, i.e. what about current consumption?

Unfortunately, we do not have any concrete way of reproducing this, and also we have no modem trace available for you, as this is happening to devices out in customer use.
What we have is this error messages from the SDK code.

Any help is appreciated.

BR

-Alf

Related