nRF9160 on a custom board having modem crash, reason 0x10

[Alfie321]

We have this custom sensor board, running nRF9160.

Application code built with SDK 2.5.0, running modem FW 1.3.2, and using LTE-M connecting to MS Azure IoTHub with MQTT.

Mostly, our asset tracker application runs just fine, but in rare cases the modem calls starts to fail, and the SDK error log reports that the modem has crashed.

This is (the essence of) our error log when the error occurs:

[34:31:47.742,065] <err> nrf_modem: Modem has crashed, reason 0x10, PC: 0x119f0a
[34:33:17.807,312] <err> azure_iot_hub: mqtt_helper_connect failed, error: 11
[34:33:17.807,342] <err> cloud_handler: Failed to connect to Azure IoT Hub, error 11
[34:33:17.917,999] <err> lte_lc: Failed to set functional mode. Please check XSYSTEMMODE.
[34:33:17.918,029] <err> cloud_handler: Failed to disconnect from the network (Connect mode 2, PSM mode 0), error -14
[34:33:20.918,151] <err> lte_lc: Failed to set functional mode. Please check XSYSTEMMODE.
[34:33:20.918,182] <err> cloud_handler: Failed to disconnect from the network (Connect mode 2, PSM mode 0), error -14
[34:33:23.918,304] <err> lte_lc: Failed to set functional mode. Please check XSYSTEMMODE.
[34:33:23.918,334] <err> cloud_handler: Failed to disconnect from the network (Connect mode 2, PSM mode 0), error -14
[34:33:26.918,579] <err> lte_lc: Failed to set functional mode. Please check XSYSTEMMODE.
[34:33:26.918,609] <err> cloud_handler: Failed to disconnect from the network (Connect mode 2, PSM mode 0), error -14

This might happen after a few days of running, after several weeks, or just (mostly) never happening.

As you might see from the logs the first error=11  is returned when trying to connect from the call to lte_lc_connect_async().

Of course there are a lot of connect calls performed earlier that works just fine, but here the modem seems to have crashed and causing the error.

1. Is is possible to explain anything regarding the crash reason (being 0x10) reported in the error message?
2. Is is possible to recover the modem after such a crash, or is a reset the only way?
3. What is the state of the modem (or SIP) after such a crash, i.e. what about current consumption?

Unfortunately, we do not have any concrete way of reproducing this, and also we have no modem trace available for you, as this is happening to devices out in customer use.
What we have is this error messages from the SDK code.

Any help is appreciated.

BR

-Alf

[Øyvind]

Hi Alf, 

Very difficult to tell exactly what is causing this, as I could not find anything directly related to the PC. This is definitely something from the modem which will need a modem trace to be 100% sure. 

But I have forwarded the first line to our modem team and hope to get something to help pinpoint the issue. 

Is is possible to recover the modem after such a crash, or is a reset the only way?

Reset is most likely the only way to recover, but hope to have more answers from our team by end of tomorrow. 

Kind regards,
Øyvind

[Øyvind]

Hi Alf, a small update. The modem team will have a look into this and see if they can find anything. 

Of course there are a lot of connect calls performed earlier that works just fine, but here the modem seems to have crashed and causing the error.

Are you able to tell how far in the application this happens? I.e. what calls were performed before this? Anything that can help our team find the culprit.

Kind regards,
Øyvind

[Øyvind]

Alf, are you able to test with modem FW 1.3.6 on any of your devices? 

-Øyvind

[Alfie321]

Thank you Øyvind!

We might do some tests with MFW 1.3.6 on some devices, but on our "out in field devices" currently running 1.3.2 this is not so simple.

What we find from your mfw compatibility matrix, the 1.3.6 needs SDK 2.5.1.
How strict is this?

BR

Alf

[Håkon Alseth]

Hi Alf,

 

My apologies for the late reply.

To be explicit about the modem assert that you originally observed: This is related to a corner case with high priority search, which we addressed with mfw v1.3.6.

Alfie321 said:

What we find from your mfw compatibility matrix, the 1.3.6 needs SDK 2.5.1.
How strict is this?

Not strict. There is no LTE related changes between ncs v2.5.0 and v2.5.1 (see here: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/releases_and_maturity/releases/release-notes-2.5.1.html), so I would say that this is a oversight from our side, which I will report internally.

That being said, we always recommend that you test your devices thoroughly before doing any form of FOTA upgrade, in case there are any problems with the overall process.

 

Let me know if you run into any issues or have any questions.

 

Kind regards,

Håkon