Modem network notifications delayed

[kuchx]

Hello,

I am working on nRF9160 custom board. Since I am reprogramming a lot, this issue is kind of getting on my nerves. Here is the issue:

Usually modem takes couple of seconds after initialization to turn on the RRC module and create a connection to network. But sometimes after some failed connections to server or just reset, the modem takes multiple minutes to receive the RRC notification and its getting frustrating. Once it was like 14 minutes, I tried reprogramming FW, downgrading and upgrading modem FW, but nothing really helped. And then after couple of times waiting for the connection to establish, on the next reset it started to work fine again. In past 2 weeks this already happened twice - once it took 4 minutes to connect and since yesterday it takes 12 minutes to receive the RRC notification and establish connection with network. I have not changed my location so the Cell is the same every time. The Cell ID notification is received within seconds since startup but the RRC for some reason takes super long time. 

I tried searching through already existing tickets but I can't really relate to those issues, some of them mention operator change or cell change, but that for me is stationary. So for obvious reasons I think the modem for some reason went rogue and I still haven't found a way to properly recover it. Since our FW currently waits around 30 seconds to get network connection, means that when modem in this state will never connect to network and device is pretty much useless. 30 seconds because its a battery powered device and usually that is enough for the modem to connect to network.

So my questions are:

1) Is this something you are aware of?

2) Why the modem re-flashing does not help?

3) Since re-flashing does not work, is it because the modem HW is somehow damaged but then again why does it bounce back to normal operation?

4) How I can properly recover modem in this situation?

5) Is the same issue observed on nRF9151 SiP?

I am currently using 1.3.7 modem version and if it makes any difference working on 2.6.0 SDK. Operating in LTE-M

Regards,

Andris

[Benjamin]

Hi Andris,

Are you able to do a modem trace and upload the .mtrace file here?

Best regards,
Benjamin

[kuchx]

Hello Benjamin,

The issue reappeared and I managed to get a modem trace. It took about 24 minutes before MCU received the RRC connected notification from modem, but as before it took a couple of seconds since startup to receive Cell ID notification. 

Regards,

Andris

[Benjamin]

Hi Andris,
The issue is caused by the reset loop restriction, which can occur if the device is flashed frequently.

To prevent this, put the modem into offline mode before flashing or powering off the device. You can also reset the reset loop counter using the AT%XFACTORYRESET command.

Best regards,
Benjamin

[kuchx]

Hi Benjamin,

I was aware of this reset loop protection but I am not really convinced that this is actually the case. If it actually would be reset loop protection, then I would expect that after successfully creating connection in the given trace, at the next reset device would immediately attach to network for the next at least 6 reset times, since the timer was expired. But that was not the case.

I noticed this also before when the RRC connected notification took about 7 minutes to be received - I waited for the attach, device worked for a while, restarted it and again I would have to wait the same timeout. 

Another question - if I remove power from the device, the reset loop counter froze's (at least that's what I expect)? In some cases the issue appeared at the end of the work day and I shut it down, came back the next morning and issues still persisted.

Anyhow I still don't believe that the issue here is reset loop counter regarding the very first paragraph of this answer. Hope this gives you some more insights about the issue. Or maybe I don't fully understand the working concept of the reset loop counter? Let me know what are your thoughts on this

Best regards,

Andris

[Benjamin]

Hi,

The modem team reviewed your trace and confirmed that the issue is caused by the reset loop restriction.

kuchx said:

But that was not the case.

Did the modem run for over 60s after initialization without any resets?

The counter is reset if the modem reinitializes with +CFUN=0 before the limit is reached, or if no reset occurs within 60 seconds after initialization. 

If you enable modem domain events with AT%MDMEV=1, the modem will send a %MDMEV: RESET LOOP notification whenever the reset loop restriction is triggered. There is no direct way to read the reset count, but you can track it by counting how many times this notification appears.

Regarding your question: the timer does not run when the modem has no power. It resumes once the modem has been initialized.

Best regards,
Benjamin

[kuchx]

Thanks for the quick response!

Benjamin said:

Did the modem run for over 60s after initialization without any resets?

I cannot confirm completely at the moment. I think I waited at least a minute before reset but I might be wrong.

I'll try you suggested solutions when I get the device in this state again. For now, thanks for your help and I'll get back to you how it goes

Best regards,

Andris

[kuchx]

Some follow up questions regarding reset loop restriction

In case of a battery powered device, when the batteries are dying the device might start to reset rapidly causing reset loop restriction to kick in. After batteries are changed device need to initialize modem and wait 30 minutes. 

1) After that device needs to be in active (RRC conneted) state for at least 60 seconds? 

2) Does it mean the modem can't be put in idle mode during this timeout?

As a test I just waited until the timeout runs out, modem attached to network, sent its data and went to idle mode, and then I waited for additional 2 minutes before i restarted the device but that did not seem to clear the reset loop restriction.

Regards,

Andris

[Benjamin]

Hi Andris,
The reset loop restriction is a mechanism to prevent signaling when something has gone really wrong repeatedly. Normally, you should never get into the reset loop restriction. 

1. No, after the device has waited 30 minutes, the modem starts normal behavior and automatically regains LTE service for possible connections the same way as when the modem is activated without the reset loop restriction.
2. No, it does not matter. If the modem is activated after initialization and registration to network succeeds before 60 s has passed, the 60 s validation timer is restarted. When the timer expires, the counter is reset.
   
   

kuchx said:

but that did not seem to clear the reset loop restriction

Did you try AT%MDMEV=1?
Could you provide new traces with this, and capture what you described here?