Modem 30 minute lockup

RE: Modem 30 minute lockup

Achim Kraus — Sat, 20 Jul 2024 07:05:42 GMT

[quote userid="89447" url="~/f/nordic-q-a/113011/modem-30-minute-lockup/494916"]If you have examples of multi-imsi-SIM-cards with auto-switching, please let me know that. [/quote]

I've tested a couple of them.

Some switch after 2 minutes, which may cause larger issues, especially, if both NB-IoT and LTE-M should be considered in the network search. E.g. in a region with LTE-M only a modem with an config "NB-IoT preferred" and 2 minutes network search timeout will hardly work. To escape that you need to detect the imsi switch (by reading the IMSI and compare it with the last one). If it happens, it may be a good idea to swap the preference (here from NB-IoT to LTE-M) after 6-10 minutes network search without success.

Some others switch after 10 minutes. Here you need some patience. E.g. if there are 3 imsi on the SIM, it may take 20 minutes to have the right one active.

Some of the SIM card providers offer also SIMs, where it's possible for the App to select one of the available imsis. That helps a lot, if you know, which one is for your region and how to switch it.

RE: Modem 30 minute lockup

Achim Kraus — Sat, 20 Jul 2024 06:58:58 GMT

The easiest and fastest way is just to check the IMSI frequently in the app. You may also ask your SIM card provider, but that may take more time and is sometimes not that easy.

RE: Modem 30 minute lockup

Achim Kraus — Sat, 20 Jul 2024 06:57:10 GMT

From the nrf9160_at_commands_v2.4.pdf, page 72.

"The proprietary %XFACTORYRESET command resets the modem to factory settings. This operation is
allowed only when the modem is not activated."

Therefore you need AT+CFUN=0

RE: Modem 30 minute lockup

Burt — Sat, 20 Jul 2024 04:14:30 GMT

Oyvind,

I want to correct my statement that AT%XFACTORYRESET=1 was the "trick" to have the Soracom SIM + nRF9160 combination register quickly with the network. What sped up the registration to under 10 seconds was the AT+CFUN=0 that I preceded the XFACTORYRESET with.

Having said that, please help me understand why issuing AT+CFUN=0 prior to power resetting my nRF9160 is correcting "something." And note that I only have to issue the +CFUN=0 once; then I can do multiple power resets with quick registrations (although I did not have the system powered on for more than a few minutes at a time in this testing).

Burt

RE: Modem 30 minute lockup

Burt — Sat, 20 Jul 2024 03:59:44 GMT

Thanks, Achim, yes, this is good to know. If you have examples of multi-imsi-SIM-cards with auto-switching, please let me know that. We use a Twilio Super SIM which is multi-imsi; I guess it is auto-switching, but I don't know when I should expect the imsi to auto-switch. We also use Soracom SIMs; I don't know whether or not those are multi-imsi.

RE: Modem 30 minute lockup

Achim Kraus — Fri, 19 Jul 2024 20:15:34 GMT

Sometimes long network searches may be caused by multi-imsi-SIM-cards with auto-switching. You may check that by reading the IMSI (AT+CIMI) frequently, e.g. every minute. If it changes, the modem usually restarts the network search. It's only one cause, but sometimes a nasty one. Even if it doesn't apply in your case, it's good to know it.

RE: Modem 30 minute lockup

Burt — Fri, 19 Jul 2024 18:58:09 GMT

Thanks, Oyvind. Yeah, I guess it's unfortunate that I opened the ticket with the "30-minute lockup" title. This was only one issue that arose while trying to work with Twilio SIM cards.

I think I had a few extra questions for you. The one that sticks out in my mind: will AT%XFACTORYRESET=0 wipe out my certificates? In case I want to use it for clearing a reset loop, I want to know how much pain to expect.

Oh, more questions, I saw the NS white paper about the reset loop restriction. I found that they did not define clearly what is meant by modem initialization and modem reset. I usually think of modem initialization in terms of the software (lte_lc.c). Reset refers to power reset, soft reset, WDT reset, and that class of events? Or is there something that can reset the modem that happens independently of a full nRF9160 reset and I won't see in the log? [Maybe dumb questions but because I cannot see when the counter is climbing up to 7 I don't know what got me there.]

Thanks, Oyvind.

Burt

RE: Modem 30 minute lockup

Øyvind — Fri, 19 Jul 2024 10:30:12 GMT

Hi Burt, I apologize for not being fully switched on, you are right in regards to the EMM events. I was too focused on the modem reset issue that I overlooked these messages in the modem trace.

I'm not familiar with the Twilio SIM, I think you will need to reach out to your local sales rep for more information on this. The SIM must support LTE-M and NB-IoT, but from what I understand only US based NB-IoT is supported.

RE: Modem 30 minute lockup

Burt — Thu, 18 Jul 2024 23:45:27 GMT

Another thing I find a bit strange: in the pdn library, they enable CNEC_ESM events to appear in the log but not CNEC_EMM events. Here is what I get if I enable simple printing of EMM events:

+CEREG: 2,"F460","00F536C9",9
[00:00:05.461,456] <dbg> modem_module: lte_evt_handler: LTE cell changed: Cell ID: 16070345, Tracking area: 62560
[00:00:05.461,608] <inf> app_event_manager: MODEM_EVT_LTE_CELL_UPDATE
+CSCON: 1
[00:00:05.527,862] <dbg> modem_module: lte_evt_handler: RRC mode: Connected
+CNEC_EMM: 6

+CNEC_EMM: 6
+CEREG: 0
[00:00:06.663,574] <dbg> modem_module: lte_evt_handler: LTE cell changed: Cell ID: -1, Tracking area: -1

You see, even if I gray out the debug level messages, I mean, if you imagine they are not present, I would have seen really quickly that an EMM 6 happened. A quick perusal of the AT Commands Reference and following their suggestion to look it up in 3GPP 24.301 would show me quickly that using this SIM, the network is rejecting my IMEI (turns out KORE Wireless support informed me that Twilio only accepts the TAC for old B0 devices; I am using B1).

Burt

RE: Modem 30 minute lockup

Burt — Thu, 18 Jul 2024 19:14:57 GMT

Oyvind, Hi, I am surprised that you didn't have me issue an AT+CEER command! It is showing an "EMM 6" error, which I understand is a complaint from the network about an unacceptable IMEI.

And, if we went through all the trouble to do a modem trace, wouldn't it know about the EMM 6 problem regardless of whether I manually issued the +CEER command?

The same IMEI is fully accepted when I use a Soracom SIM. So T-Mobile must have no problem, I would think. Is it normal that the home Twilio network would check the IMEI? Any thoughts you have are appreciated.

Burt

RE: Modem 30 minute lockup

Burt — Thu, 18 Jul 2024 01:22:11 GMT

A couple more questions:

Related to %XFACTORYRESET=0, will this erase my loaded certificates put in via %CMNG or will it not touch them? Too bad there are not more details in the AT Commands Reference. Speaking of which, it would be nice if it stated explicitly in that document that reading certificates with %CMNG will not work when something like asset_tracker_v2 is running; it can get confusing when list works but read does not, etc. (It becomes harder and harder to search for or remember "getting started" type documentation when working with AT Commands Reference, if you follow my drift.)

And here is an excellent question: while doing nothing more than switching application firmware back and forth between NB-IoT and LTE-M, and running them, given a SIM from Soracom, and running in NB-IoT mode, we find it often takes about over 4 minutes to connect to the T-Mobile network (this is not the Twilio SIM that refuses completely to connect). But if I issue an AT%XFACTORYRESET=1 properly, and then power reset, the board connects in under 10 seconds. And it continues to connect quickly given a few power resets. Please help me understand what garbage is getting stuck in the modem that needs this cleaning. It's a really cool discovery!

Burt

RE: Modem 30 minute lockup

Burt — Wed, 17 Jul 2024 14:29:12 GMT

Thank you very much, Oyvind. I do have a couple of questions.

You say that nothing out of the ordinary shows up in the modem trace. But what I see using the Cell Monitor Dashboard or looking at log messages is that the modem stays in searching, not registered mode, and never gets to registered mode. Any hints from the trace about why that is? The other SIMs we have, from Soracom, do not have that issue; they register.

Question about the %XFACTORYRESET=0 command: I was afraid to try that (as opposed to %XFACTORYRESET=1); I didn't know if I would have to reinstall modem firmware or something more drastic. Can you give me a few words about this? I wish also that the "few words" could be added into the AT Commands Reference for those of us who have limited expertise to understand "all modem data".

Thanks, Oyvind.

Burt

RE: Modem 30 minute lockup

Øyvind — Wed, 17 Jul 2024 14:09:14 GMT

Hi Burt, thanks for sharing the modem trace. I was able to open but did not find anything out of the ordinary. The modem reset loop happens especially in the field when roaming. Have you read the Modem Reset Loop Restriction white paper? The modem reset loop restriction serves as a safeguard against unexpected issues with the cellular modem and application. Its purpose is to protect networks in case the application or cellular modem does not behave as expected. This protection ensures that the nRF91 device does not congest the network, for example, with excessive signaling (Attach requests), which could occur if the application and modem continuously reset and attempt new connections in a loop.

The modem counts all resets where the modem is not gracefully deinitialized with +CFUN=0. If modem deinitialization with +CFUN=0 is triggered before the predefined limit is reached, the reset counter is set to zero and the counting starts from the beginning. In modem firmware version 1.3.0, the limit is 5. In modem firmware version 1.3.1 and later, the limit is 7.
In product development phase, active reset loop restriction can be removed with the %XFACTORYRESET=0 command. The command should not be used for this purpose in the final product. If %XFACTORYRESET=0 is used, all modem data, including user-configurable data, is reset.

Kind regards,
Øyvind

RE: Modem 30 minute lockup

Burt — Wed, 17 Jul 2024 02:03:48 GMT

devzone.nordicsemi.com/.../trace_2D00_2024_2D00_07_2D00_17T01_2D00_04_2D00_30.325Z.mtrace

Can you try this trace, Oyvind? I am seeing a MODEM RESET LOOP in the Cellular Monitor Dashboard. Also, I added a line of code

err = nrf_modem_at_printf("AT+CGDCONT=%u, %s, %s", 1, "Non-IP", "iot.nb");

Similar to what I mentioned earlier. I tried various flavors of "IP" "IPV6" "IPV4V6" and the "Non-IP" shown here. None solve a problem but do make a PDN appear in the Dashboard.

RE: Modem 30 minute lockup

Øyvind — Tue, 16 Jul 2024 04:59:33 GMT

Hello Burt,

[quote user="Burt"]but it does not solve that Twilio/T-Mobile/NB problem.[/quote]

I will need to see the modem traces in order to see what is going on inside the device. Do you get any error messages in your application logs?

Thanks!

Kind regards,

Øyvind

RE: Modem 30 minute lockup

Burt — Sun, 14 Jul 2024 22:53:48 GMT

Okay, I figured out how to fix the Soracom SIM problem: AT%XFACTORYRESET=1 after AT+CFUN=0. That got rid of whatever hidden garbage I had inadvertently added.

And that helped me further confirm that the Twilio and T-Mobile present Narrowband SIM is a mess.

I changed the CGDCONT code above to

err = nrf_modem_at_printf("AT+CGDCONT=%u,%s,%s", 1, "IP", "iot.nb");
err = nrf_modem_at_printf("AT+CGDCONT=%u,%s,%s", 2, "IP", "wireless.twilio.com");
err = nrf_modem_at_printf("AT+CGDCONT=%u,%s,%s", 3, "IP", "super");

which seems to not hurt anything but it does not solve that Twilio/T-Mobile/NB problem.

Burt

RE: Modem 30 minute lockup

Burt — Sun, 14 Jul 2024 19:07:21 GMT

Now I am not sure of anything. It seems that I now have killed a perfectly good Soracom SIM card by inserting a line of code into lte_lc_func_mode_set() [just a convenient place]

err = nrf_modem_at_printf("AT+CGDCONT=%u,%s,%s", 0, "IP", "soracom.io");
if (err) printf("setting CGDCONT returns %d in %s\n", err, __func__);

The SIM will no longer connect. If I use a good SIM I connect and the command

AT+CGDCONT?

returns

DCONT: 0,"IP","soracom.io","10.164.75.168",0,0

after connecting. The bad SIM (that was a good SIM before experimenting with AT+CDGCONT) returns

+CGDCONT: 0,"IP","soracom.io","",0,0

How did this mess occur and how do I fix it?

Thanks,

Burt

RE: Modem 30 minute lockup

Burt — Sat, 13 Jul 2024 03:50:31 GMT

I guess I misread that Twilio web page. I thought they were telling me that I needed to configure the APN somewhere in order to connect/register to the network; like I had to copy the APN that the SIM uses to somewhere else in order to establish an NB-IoT connection. Anyway, I have a ticket open with KORE Wireless/Twilio and they opened a secondary ticket with T-Mobile. I don't think those KORE Wireless folks have any clue about the company Twilio that they purchased; not on a technology level, at least.

RE: Modem 30 minute lockup

Burt — Fri, 12 Jul 2024 22:39:41 GMT

I eventually was able to get out of the mess. It began when I tried adding this to asset_tracker_v2 prj.conf

CONFIG_LWM2M_CARRIER_PDN_IPV4=y
CONFIG_LWM2M_CARRIER=y
CONFIG_LWM2M_CARRIER_SETTINGS=y
CONFIG_LWM2M_CARRIER_CUSTOM_APN="iot.nb"

in an attempt to do the equivalent of

AT+CGDCONT=1,"IP","iot.nb"

as indicated by the clowns at https://www.twilio.com/docs/iot/wireless/network-configuration#how-do-i-set-the-apn-with-at-commands. Not only did I lock up the modem but I wasn't configuring any "iot-nb" because I don't know how to set auto_startup to 1 in lwm2m_settings.c. It involves a modem key but I have no idea how to create a simple modem key appropriate for the purpose. It seems like this is another complicated topic involving security knowledge. Please help.

I also tried setting

CONFIG_PDN_DEFAULTS_OVERRIDE=y
CONFIG_PDN_DEFAULT_APN="iot.nb"
CONFIG_PDN_DEFAULT_FAM_IPV4=y
CONFIG_PDN_LOG_LEVEL_DBG=y
CONFIG_NVS=y
CONFIG_MODEM_KEY_MGMT=y
CONFIG_SMS=y

and although that did not break anything, it did not solve the problem whereby Twilio SIMs will not connect to T-Mobile. Can you tell me why they cannot just keep things simple like Soracom and the rest of the world?

I guess you have people who know all the tricks and I hope you can help me get back on my feet again. Thanks. Oh, yeah,

Burt