nrf9160 modem lockup after 24 hours, occasional random resets

RE: nrf9160 modem lockup after 24 hours, occasional random resets

Hakon — Wed, 18 May 2022 11:20:29 GMT

Next release should be public during this week. MFW 1.3.2.

RE: nrf9160 modem lockup after 24 hours, occasional random resets

amessner — Thu, 12 May 2022 20:17:36 GMT

Also, I had mentioned that I've been seeing random resets:

These random resets wouldn't show on certain boards at all, but on some almost once an hour.

After enabling the logs in the SPM and disabling the automatic resets on hard fault (CONFIG_RESET_ON_FATAL_ERROR) I finally got some insight on where those random resets are coming from. The AT command thread causes a hard fault.

I've been using nRF Connect SDK 1.8.0 and the AT command interface (CONFIG_AT_CMD).

After looking through the documentation I noticed that the AT command interface is deprecated and one should use the AT interface API instead. So I updated the nRF Connect SDK to 1.9.1 and switched to the AT interface API and the resets went away.

Anyways, I just wanted to add this info in case it helps someone...

RE: nrf9160 modem lockup after 24 hours, occasional random resets

amessner — Thu, 12 May 2022 20:01:30 GMT

Thanks for looking into this Hakon. What is the timeline for the next modem release?

RE: nrf9160 modem lockup after 24 hours, occasional random resets

Hakon — Wed, 11 May 2022 09:24:47 GMT

Yes, the modem team has confirmed there will be a fix next release.

RE: nrf9160 modem lockup after 24 hours, occasional random resets

amessner — Thu, 05 May 2022 14:55:27 GMT

Any updates on this?

RE: nrf9160 modem lockup after 24 hours, occasional random resets

Hakon — Sun, 01 May 2022 11:14:33 GMT

Hello,

I will have the modem team look at this.

RE: nrf9160 modem lockup after 24 hours, occasional random resets

amessner — Fri, 29 Apr 2022 15:17:35 GMT

Just adding a bit more information:

We tried different SIMs (eSIM and SIM cards) of different network operators, but the issue didn't go away.

I parsed the last 3h of the modem trace I attached using the Trace Collector V2:

I marked the position where the problem starts. There should be a TLS handshake after the successful TCP handshake, but the nrf9160 drops the TCP connection and sends a FIN 10s after the successful TCP handshake with our server. Could this be a potential TLS (TLS sockets are offloaded to the modem) issue?

After that there are a bunch of RCCConnectionReconfiguration events and finally a RCCConnectionRelease.

Then the connection gets re-established. While all this re-configuration/re-establishment is happening the application carries on (not being aware of the connection issues) and obviously keeps failing to talk to the server. It finally tries to disconnect the LTE network, but then something goes wrong - half of the usual disconnect messages are missing: