nRF9160 goes offline and never comes back online

Hi

We currently have products in field test that disconnects from network and never reconnects. It only occurs to a few devices and not all. Restarting the device works.

Devices are below a manhole cover so RSSI is not the best but not to bad either. We report data each minute to a cloud and RSSI has been e.g. -110 when a device went offline.

Device is setup to use NB-IOT as fallback if LTE-M fails.

Modem firmware is 1.3.5 as this matched our ncs version at development. We will try to use 1.3.7 on new devices and see if they will fail.

We have tried to simulate bad network conditions so device keeps disconnects/reconnect by removing the antenna and using shielding but we were not able to make the device fail to go online again.

We have been in contact with our cellular provider and hopefully you might see what could be wrong based on what they have reported back to us.

At first they tried to disable the device from network and reenabled. This according to them brought the device back online for 5 seconds before the following (I suppose they look at core) happened:

INITIAL CONTEXT SETUP, ACTIVE, CONTEXT RELEASE REQUEST (User-inactivity)

This pattern repeats, NO UL/DL, UE has nothing to say.

Another time the device went offline, they gave the following information:

Looking at the time when the device went offline, the following sequence occurred:

22:40 Change eNb

22:42 Re-attach (I see no dettach to eNb) 

22:42 Delete session / create session to new eNb

22:43 One DL notification to UE, but ”UE not responding”

23:13 Tries again to gain contact where UE pages in whole TAC but  ”Unable to page UE”.

They also provided a GTPv2 trace (see attached)

We also received the following images when again asking if they could see why device went offline.

Not sure if related as the device succeed to connect a few minutes later.

More interestingly might be that they can see that device switch between using LTE-M and NB-Iot.

Another time they report that device request release from network and networks never sees it again.

20260125

2026-01-25 08:11:37

238023637022350

4520873275

internet.mnc002.mcc238.gprs

CRR_RADIO_BEARER_RELEASE_DUE_TO_ENB_UE_CONTEXT_RELEASE_REQUEST

20260125

2026-01-25 09:25:37

238023637022350

4520873275

internet.mnc002.mcc238.gprs

CRR_MOBILE_REACHABLE_TIMER_EXPIRY

We have tried to implement a watchdog timers that shutdown and reenable the modem if no data has been delivered for a certain time and device "thinks" it is online (having a connection to azure IOT cloud) but this has not helped. We have seen that modem sometimes fail to respond on AT commands, so we implemented software reboot if device fails to shutdown and reenable modem.

Hopefully based on the following information you can help on what could be wrong or next steps to debug.

4428.GTP_V2_Dialogue-20250509_MSISDN_22176521.pcap

Parents Reply Children
Related