nRF9160 fails to attach after several RESETs & attach without power cycle

[Michal Mühlpachr]

Hi,

nRF9160 with SLM NCS v1.9.1 fails to attach after 7th RESET (by RESET pin) & attach without power cycle.

There is not any ERROR state reported by modem, modem fails to attach forever without power cycle.

After nRF9160 power cycle it attaches immediately with same AT sequence.

We need reliable initialization of nRF9160 SLM/Modem without nRF9160 power cycle because we do not have load power in production HW - issue threatens device stability in the field.

I prepared Python script to reproduce issue with nRF9160DK, you can find it together with log at https://gist.github.com/muhlpachr/40a4d8f1b332aee8a25fa7e1ad023751
Modify COM port for AT commands according your PC setup please.

Issue is 100% reproducible.

Attached you can find modem trace.

Thank you.

Regards,
Michal

trace-2022-03-24T23-42-16.760Z.bin

[Øyvind]

Hi Michal, 

our developers did have a look at the modem trace, and they did repspond with the last clarification. 

What are the reasons for why you are resetting the device? Our developers write:

If they want to avoid the reset loop restriction, they have to wait 60+ seconds after modem initialization before triggering the reset. This way modem's reset counter will not ever reach the limit and trigger the creation of the reset loop restriction.

Please understand that reset loop is a safety mechanism and is not supposed to be entered if the device is behaving "in a good way".

Kind regards,
Øyvind

[Michal Mühlpachr]

The reason why we are resetting nRF91 is development cycle.

In our HW nRF91 with SLM is controlled by host MCU nRF52. When nRF52 is flashed and new version of our application starts to execute, it needs to transit nRF9160/SLM in guaranteed known state at startup, so it resets nRF91 during startup by pin.

We suppose it is not wise to start nRF91 initialization by +CFUN=0 because:
1. it would store plenty of undefined configuration in nRF91 NVM
2. since nRF91 NVM is implemented in flash, it would wear nRF91 flash fast during app development and tests on nRF52 (e.g. application on nRF52 can be restarted by Zephyr shell command)

It is not feasible to power-cycle our HW during app development because there are supercaps and constant current power source to protect batteries (it operate from bench PSU during development but current limit circuity is active all the time even during development - is has to, to properly test HW+SW) and those discharge-charge supercaps cycle takes time.

We would take pain and wait 60+ seconds at application startup during development cycle but it does not help to prevent RESET LOOP protection because nRF91/SLM does not behave like it is described/supposed.

Comment/info from developers is not related to collected trace in any manner (there are 120+ seconds delays after modem initialization between RESETs on nRF91), it just describes how nRF91 is supposed to behave.

It is not the case unfortunately and it is obvious from trace, I do not know how to prove it more.
Why you can not see it from trace please ?

What other development cycle do you propose to prevent RESET LOOP and guarantee proper deterministic nRF91 initialization please ?

Please understand that we do require proper and convenient development cycle and we do not know how to that because nRF91 does not behave according description from developers and modem source code is closed, so do not have any other affordable way how to find way than asking for support.

Kind regards,
Michal

[Øyvind]

Thanks for providing this clarification. I've forwarded this to our modem team. 

Kind regards,
Øyvind

[Øyvind]

Hello again, 

I've got some feedback based on the last modem trace that you provided. From this our nRF91 team has the following feedback:

Based on the log, the device does not wait 60s+ after successful Attach before triggering reset.

00:00:17.284606  NAS_PDU_ATTACH_COMPLETE_W_ACTIVATE_DEFAULT_EPS_BEARER_CONTEXT_ACCEPT [074300035200C2] 
00:00:29.912078 L23_AT_STRING AT_STRING => +CFUN=1 
00:00:37.004303  NAS_PDU_ATTACH_COMPLETE_W_ACTIVATE_DEFAULT_EPS_BEARER_CONTEXT_ACCEPT [074300035200C2] 
00:00:47.108367 L23_AT_STRING AT_STRING => +CFUN=1 
00:00:54.326721  NAS_PDU_ATTACH_COMPLETE_W_ACTIVATE_DEFAULT_EPS_BEARER_CONTEXT_ACCEPT [074300035200C2] 
00:01:04.556671 L23_AT_STRING AT_STRING => +CFUN=1 
00:01:11.758941  NAS_PDU_ATTACH_COMPLETE_W_ACTIVATE_DEFAULT_EPS_BEARER_CONTEXT_ACCEPT [074300035200C2] 
00:01:22.482543 L23_AT_STRING AT_STRING => +CFUN=1 
00:01:29.760192  NAS_PDU_ATTACH_COMPLETE_W_ACTIVATE_DEFAULT_EPS_BEARER_CONTEXT_ACCEPT [074300035200C2] 
00:01:40.499420 L23_AT_STRING AT_STRING => +CFUN=1 
00:02:05.188568  NAS_PDU_ATTACH_COMPLETE_W_ACTIVATE_DEFAULT_EPS_BEARER_CONTEXT_ACCEPT [074300035200C2] 
00:02:17.234619 L23_AT_STRING AT_STRING => +CFUN=1 

This is the reason why the reset counter keeps increasing until there has been 7 resets. At that point the reset loop restriction is activated. You need to ensure that you wait at least 60 seconds after successful Attach before triggering the reset. This is to ensure that the reset counter does not increase. 

It is also possible to clear an active reset loop restriction by factory reset.

Based on this information, our nRF91 team informs me that you need

1. Wait at least 60 seconds before reset OR
2. Count each attaches and then wait 60 seconds after 4th successful attach. (+CGATT)
3. If above is not possible, do a factory reset (%XFACTORYRESET) after the RESET LOOP hits during 7th attach attempt

Kind regards,
Øyvind

[Michal Mühlpachr]

Hello again,

this trace is not from trace-2022-03-28T21-28-11.914Z.bin unfortunately who was collected to show that even 120+ seconds does not help.

Would you be so kind and check correct trace please ?

Regards,
Michal