nrf910 DK runs into modem reset loop running Asset Tracker V2

[sebastian_]

Setup

• nRF9160-DK
• PCA10090 1.1.0 2022.4
• Modem FW: mfw_nrf9160_1.3.3
• Asset Tracker V2, I tried both the downloaded version from here: https://www.nordicsemi.com/Products/Development-hardware/nRF9160-DK/Download#infotabs (2022_12_08_188a1603) and building Asset Tracker V2 from source from the v2.2.0 release tag of the NRF Connect SDK. The described behaviour is identical.
• LTE Link Monitor v2.0.2
• Host machines running OSX and Arch Linux, both set up with the latest Zephyr toolchain and the 2.2.0 SDK release
• Followed nRF cloud and SIM card setup, the DK _is_ able to post events to nRF Cloud eventually
• I am using the iBasis SIM card provided with the kit
• I am based in Vienna, Austria, LTE coverage should not be an issue, however, NB-IoT is not expected to be working with the iBasis SIM card which may or may not be an issue as the modem seems to start to switch modes at some point if it fails to register

Problem description

1. Connect the DK
2. Launch LTE Link Monitor
3. Connect to the DK, first modem port
4. AT+CFUN? may or may not be required to get the UI to refresh, MODEM is green and up, LTE phasing from yellow to green.
   1. I get
      AT+CEREG?
+CEREG: 5,2,"36ED","0043D92A",7
      or similar.

At this point, I would wait for a while (aka 30 minutes) until the DK eventually connects and starts posting data to nRF cloud.

At some point I noticed that, when restarting the device through either one of those three ways

1. Using the "Reset Board" button in the "Connected devices" section in VS Code (I assume that triggers an nrfjprog reset?)
2. Using the Power Switch SW8 on the DK
3. Disconnecting and reconnecting the USB cable of the DK

while having LTE Link Montor attached to the modem UART, the first thing the Asset Tracker firmware logs is:

[00:00:03.479,187] [1;33m<wrn> modem_module: The modem has detected a reset loop. LTE network attach is now restricted for the next 30 minutes. Power-cycle the device to circumvent this restriction.

So, at this point I could wait for 30 minutes, after that or within the first two minutes or so, the DK connects to LTE and starts sending updates to nRF cloud. This is perfectly reproducible, but quite time-consuming. I have attached a log file demonstrating the behaviour with one power cycle in the middle, using the DK hardware power switch.

Questions

1. Why is this happening? From the documentation I assumed, that I needed to power cycle the device at least seven times without proper deinitialization.
2. What is proper deinitialization? How can I achieve that during development where I would constantly flash new firmware and reset the board?
3. I am not able to get the DK to connect, with _any_ other sample. I tried MQTT, COAP, AT Client and a few others.
   1. I assume, the same thing is happening to me, however, those samples do not seem to listen to or check for the modem reset loop so I goes unnoticed. Is that plausible? Is there a way to verify the reset thing is happening right now? The documentation tells me to subscribe to events using AT%MDMEV=1, however, that puts me in a position where I would have to wait 30 minutes to check my assumption.
   2. The other samples do not seem to deal with that situation very well, I am stuck in an infite loop of failing registration attemps, even if I leave the kit on over night while the Asset tracker V2 firmware _does_ connect every single time after exactly 30 Minutes within the first two minutes after that. Is that to be expected? What is the difference between those samples, and how can I at least achieve the robustness that the Asset Tracker firmware has?
4. How can I avoid running into this at all.

Thank you very much for your response!

Kindest regards
Sebastian

Noticable events in the attached log:

• At line 1, the Asset Tracker V2 fw is flashed and started for the first time.
• Beginning around line 264, the 30 min ban is over and the device connects successfully
• At line 781, I turned the DK off and on again using the hardware power switch SW8 on the board
• At line 1064, again after the 30 min ban, the modem connects successfully

modem_log.txt

EDIT: I had to upload the logfile again, editing my post messed up the original attachment.

[Michal]

I'm sorry, looks like the question got lost until your reply and I did not see it.

It is now back on my list and I will get to it as soon as possible, but we have been experiencing a high workload lately.

If there are any updates please let me know.

Best regards,

Michal

[sebastian_]

Dear Michal,

I managed to solve one (big part) of the problem. As it appears, "Flash" in VS Code does not trigger the modem's reset loop detector, while using the "Erase and Flash to Board" functionality does. It would be very, very, very helpful to have some information regarding this available anywhere. E.g. somewhere inside the reset loop modem docs and probably even somewhere in the getting started section of the nRF9160.

So I have to rephrase my questions:

1. Is this expected, and if so, why?
2. When being stuck inside the modem reset loop, why does asset tracker recover after 30 minutes but the CoAP sample does not? (See logs attached)

[Michal]

sebastian_ said:

I managed to solve one (big part) of the problem. As it appears, "Flash" in VS Code does not trigger the modem's reset loop detector, while using the "Erase and Flash to Board" functionality does. It would be very, very, very helpful to have some information regarding this available anywhere. E.g. somewhere inside the reset loop modem docs and probably even somewhere in the getting started section of the nRF9160.

That's interesting, I will forward that to the developers. I assume that there is an inherent difference in erasing sectors vs erase all (which is the difference between those two functions).

sebastian_ said:

1. Is this expected, and if so, why?

I am unsure if it is something internally in the nRF9160 or if it's an IDE functionality difference, I will have to investigate that.

sebastian_ said:

2. When being stuck inside the modem reset loop, why does asset tracker recover after 30 minutes but the CoAP sample does not? (See logs attached)

I don't see any log for the CoAP sample, could you try to attach it again?

Best regards,

Michal

[Michal]

Hello,

So the difference between "Flash" vs "Erase and Flash to Board" is that they run different nrfjprog commands.

Flash runs --sectorerase and Erase and Flash to Board runs --chiperase.

The difference is that --sectorerase does not erase UICR, but --chiperase does, which is probably what triggers the modem loop detector.

From the documentation:

If --chiperase is given, all the available user non-volatile memory, including UICR, will be erased before programming. If --sectorerase is given, only the targeted non-volatile memory pages, excluding UICR, is erased.