Sim card issues (CEREG 90), possibly due to freezing weather

RE: Sim card issues (CEREG 90), possibly due to freezing weather

Sebastiaan Merckx — Thu, 02 Feb 2023 08:30:14 GMT

Just for completeness for anyone who might be interested:

we have not been able to reproduce the issue in the freezer
Someone has suggested that the issue might have been triggered (indirectly?) by a reset of the cell tower. The Mobile operator has suggested to do a "cancel location" of the sim card, but this was all perfectly captured by the modem firmware and higher-layer firmware so that is also not a trigger (it would be very surprising though, since the modem and sim card are unpowered during a reset).

RE: Sim card issues (CEREG 90), possibly due to freezing weather

Håkon Alseth — Fri, 23 Dec 2022 11:57:36 GMT

Hi,

[quote user="Sebastiaan Merckx"]

I don't handle the "CEREG: 90" notification specifically, but I have more generic recovery scenarios (e.g. first connection attempt can take 1 minute, then 4 minutes, then 7 minutes, with a "lte_lc_offline" and a "lte_lc_deinit" in between, and cold system resets if there is still no connection after the 3rd attempt.

So I'm pretty sure that these devices have tried hundreds of times. You are right that it could help to implement a specific detection mechanism to get a better view of how often this occurs in the field (on other devices). I'll put that on low prio though.

[/quote]

Ok, that is good to know! What's important is that you are able to recover from the scenario, and it sounds like you have good control over that part!

[quote user="Sebastiaan Merckx"]

I also have a device running for almost 24 hours in the freezer now (yeah, seriously ) and syncing every 5 minutes which works fine so far. It's not one of the faulty 2 units though, so that can be another test we could plan.

For now, I don't think we can make a lot of progress without reproductions, and I think it's best to close the ticket (or put it on hold) with the assumption that a mechanical issue causes the problem.

[/quote]

It does sound like you're on the right track for testing.

Feel free to update this case at any point, if you have any questions or any new behavior/info pops up.

Happy holidays!

Cheers,

Håkon

RE: Sim card issues (CEREG 90), possibly due to freezing weather

Sebastiaan Merckx — Thu, 22 Dec 2022 13:11:43 GMT

Hello Håkon,

I also have a device running for almost 24 hours in the freezer now (yeah, seriously 🙂) and syncing every 5 minutes which works fine so far. It's not one of the faulty 2 units though, so that can be another test we could plan.

RE: Sim card issues (CEREG 90), possibly due to freezing weather

Håkon Alseth — Thu, 22 Dec 2022 12:35:22 GMT

Hi,

Thank you for providing such detailed information. I just wanted to make sure that the battery was able to provide sufficient peak outputs, even at lower temps.

Based on the battery description and specification, it should have no problem driving the nrf9160, even at lower temperatures.

[quote user="Sebastiaan Merckx"]Both devices had exactly the same problem (CEREG:90), the second one was even fixed by just opening and closing the sim socket (not explicitly taking out the card) followed by a reset.[/quote]

thank you for confirming.

Does your firmware handle the "CEREG: 90" notification? What it could do is to catch this scenario, and try again x minutes later. This is however not a full fix (which would be to check or replace the SIM holder), but a way to detect the scenario.

Kind regards,

Håkon

RE: Sim card issues (CEREG 90), possibly due to freezing weather

jo_elektro — Wed, 21 Dec 2022 13:21:32 GMT

Hi,
That is my department :).
The NRF is fed by two battery's, one is a hefty 2S LIPO of about 37Wh and the second is a smaller 1S LIPO which takes over in case of removal of the Main battery.
Both of them go through a LTC3621 step-down and that 3V3 output is stabilized by a 1F capacitor Digikey to compensate for heavy current dips caused by a small DC motor further down the line.
Capacitors on the NRF have been designed as per datasheet and equal to the eval board.

RE: Sim card issues (CEREG 90), possibly due to freezing weather

Sebastiaan Merckx — Wed, 21 Dec 2022 13:16:26 GMT

Hi Håkon,

I'll let our HW engineer answer on the questions about battery type and supercaps, but I already can tell you that the powering circuit is not that easy. Main battery is https://www.rrc-ps.com/power-management/batterieladegeraete/produkt/RRC2130, so dual-cell Li-ion, 4170 mAh. There's a backup battery (single-cell, 1800 mAh) but the backup battery did not have to take over during most of the 1-week outage.

Both devices had exactly the same problem (CEREG:90), the second one was even fixed by just opening and closing the sim socket (not explicitly taking out the card) followed by a reset.

Given your information and Torje's shared experience above, a mechanical issue is indeed more likely (possibly caused indirectly by the cold?)

RE: Sim card issues (CEREG 90), possibly due to freezing weather

Håkon Alseth — Wed, 21 Dec 2022 12:59:22 GMT

Hi,

[quote user=""]1 of the 2 units resumed some communication on december 19th, but not with the correct intervals (so it definitely still had some trouble). I went on site, and noticed LTE connect errors pointing to "+CEREG: 90" which means issues with the UICC (SIM card). Several manual resets (triggered via the reset line, not by unpowering the system) did not solve the issue. Eventually, I took out the sim card from the slot for a second, and put it back in and it all started working fine.[/quote]

CEREG: 90 points to communication problems with the SIM itself.

This sounds like a mechanical issue with the SIM card holder itself. Have you tried replacing the SIM holder to see if this has an effect on the stability over temperature?

[quote user=""]Is it possible that the SIM cards or another part of the electronics get stuck after e.g. a load of invalid commands/responses, and that a power cycle is required to fix the issue? (I have to verify if the 1.8V effectively goes down on a reset).[/quote]

The supply voltage for the SIM goes off after a given time-period after enabling eDRX or PSM, depending on what your SIM supports (if it supports power down).

[quote user=""]I'm using Modem FW 1.3.0 and SDK 1.7.1, which are certainly not the latest ones. Would it make sense to upgrade these?[/quote]

I do not think that this is a root-cause of the problems you're seeing.

[quote user=""]Is it possible to reset the sim card via a firmware command? Or would that be done intrinsically on e.g. lte_lc_offline and "AT+CFUN=0"?[/quote]

If you use PSM/eDRX actively in your application, the SIM communication will only occur on-demand, meaning that the SIM will effectively be reset for each time you communicate with it.

[quote user=""]Since it's clearly not easy to reproduce this issue, have there been any similar issues or can you think of a way to reproduce the issue?[/quote]

Did both devices have the exact same problem, ie. that CEREG: 90 was logged as the failure reason?

[quote user=""]since it's battery based and includes supercapacitors so it's not that easy.[/quote]

Can you share information related to the battery type that you're using, and the supercaps on your board?

Kind regards,

Håkon

RE: Sim card issues (CEREG 90), possibly due to freezing weather

Torje — Wed, 21 Dec 2022 12:59:01 GMT

Hey,

I have several Thingy:91 here. On one of them I sometimes get the same error. Restarting the devices does not help; I have to reinsert the SIM, just like you're describing. I think its bad wiring/something mechanically faulty with the SIM-holder, as it usually occurs after handling them. Are the devices sometimes moved, or are they completely stationary? Just a thought!

RE: Sim card issues (CEREG 90), possibly due to freezing weather

Sebastiaan Merckx — Wed, 21 Dec 2022 12:55:17 GMT

I noticed that the 1.8V is brought down on hardware reset (when pressing reset button), which implies IMO that the sim card itself is not the faulty factor (it gets fully unpowered)? But that's quite contradictory to the fact that just taking out the sim card, putting it back in and doing a hardware reset solves the issue?

Not sure if a hardware reset via the reset button is equivalent to a software reset via sys_reboot(SYS_REBOOT_COLD).