Modem hangs randomly at some point

RE: Modem hangs randomly at some point

Achim Kraus — Wed, 31 May 2023 14:06:44 GMT

> but session handshake is what eats traffic.

Yes, unfortunately, there is not too much realistic information available, how frequently a TLS/TCP connection breaks. Without forecasts are very weak.

Do you measure the network searches and times, the number of cell-updates and, if applies, the number of network provider changes? All that consumes sometimes a quite a lot of energy.

Depending on the firmware version and the used SIM card, there may be also some HPPLMN searches, which, especially if unsuccessful and then repeated, also consumes a lot.

RE: Modem hangs randomly at some point

_dev_ — Wed, 31 May 2023 13:08:30 GMT

CoAP with DTLS seems interesting. I may look at that later.

At the moment I use MQTT with binary payloads, so in addition to long keep-alive there is very little overhead in the payload itself, but session handshake is what eats traffic.

RE: Modem hangs randomly at some point

Achim Kraus — Tue, 30 May 2023 07:09:07 GMT

> why TLS session cache doesn't work

Because the cloud server needs to support it. And because of some security concerns about not proper closed connection, which are still in place in many tls server implementations. The alternative, tls session ticket, seems also be considered to be deprecated (see ietf tls mailing list).

That situation is in line with my observation now for years. The TLS stuff is driven by the main usage, HTTPS, and with that, considered to be more short living. I guess, many of the MQTT server implementation didn't work on that lower TLS layer in order to overcome the TLS mainstream. And if so, they will be faced some argument, not being the mainstream.

If you go down to the ip-base, you will see, that for device communication, especially cellular, there is no alternative to non-ip or UDP. For UDP, Eclipse/Californium (I'm a developer in that project) provides a mature base to implement applications. with DTLS 1.2 CID it overcomes the most pain in cellular iot, ip-address changes, either caused by longer device sleeps or provider change. Also for CoAP/DTLS 1.2 CID it's not too easy to find a matching pair for the device and cloud, therefore I spend the last months in providing that by improving Eclipse/tinydtls and my own simple demonstration project Zephyr - coaps-client .

RE: Modem hangs randomly at some point

_dev_ — Tue, 30 May 2023 06:41:09 GMT

Thanks, I will have a look.

Network reliability and traffic control is a whole separate thing 🙂

E.g. I still can't understand why TLS session cache doesn't work all the time. It consumes extra 10kb of traffic for the handshake process, which is a public key and few other things. Over the period of a month it wastes hundreds of kilobytes, which my tiny sim plans can't easily afford. But that is a different topic 🙂

RE: Modem hangs randomly at some point

Achim Kraus — Tue, 30 May 2023 06:26:32 GMT

At least, if you detect an communication issue, you may apply a more sophisticated procedure:

1. restart only the modem

2. reconnect (that's for mqtt done implicit, I use CoAP/DTLS 1.2 CID and my encryption sessions survives the modem restart or even a network provider change.)

3. reboot

See recover procedure

In some rare cases, I also execute a modem factory reset.

(You may find also other similar issues here in the forum, e.g. discussion / doc for reliable connection missing . And some documentation - handling pdn errors on sockets .)

RE: Modem hangs randomly at some point

_dev_ — Tue, 30 May 2023 06:15:08 GMT

> Just to mention, you may reboot the device without the watchdog.

hmm, let me think if I can somehow predict such behaviour and manually reboot.

RE: Modem hangs randomly at some point

_dev_ — Tue, 30 May 2023 06:13:38 GMT

SIM card provider.

I don't see any activity on the MQTT side during the glitch.

RE: Modem hangs randomly at some point

Achim Kraus — Tue, 30 May 2023 05:18:27 GMT

> That was precisely my idea as well, to reboot SoC

Just to mention, you may reboot the device without the watchdog.

RE: Modem hangs randomly at some point

Achim Kraus — Tue, 30 May 2023 05:15:29 GMT

> data provider registers significant amount of authentication and location update events

Is that your MNO or SIM card provider? Or your MQTT server/broker?

RE: Modem hangs randomly at some point

_dev_ — Tue, 30 May 2023 04:12:43 GMT

Hi Achim,

Thanks for the ideas.

I'm pretty much using the latest stuff

B1 revision of the chip
1.3.4 version of modem firmware
2.3.0 of ncs

I looked at your implementation and it is very similar to mine, maybe because I used an example from the samples folder.

I actually have `WDT_FLAG_RESET_SOC` in flags as well 🙂.

> if I get for a certain amount of time no response from my server, I also reboot the device

That was precisely my idea as well, to reboot SoC in case if I don't get PUBACK from my MQTT provider within certain interval.

I guess it can be that at some point battery can't provide enough current so voltage drops to a dangerous level and everything becomes unstable. The only contradicting fact is that if I manually reboot, everything works again.

I will leave the device connected to my benchtop power supply and see if that is related to battery or not.

Thanks.

RE: Modem hangs randomly at some point

Achim Kraus — Mon, 29 May 2023 10:01:41 GMT

What happens, when the watchdog "bites", depends on the configuration.

Check the API docu of the watchdog for that.

Just as inspiration, here is my watchdog setup .

And just as hint:

Your nRF9160 has an hardware version and a modem firmware version. I guess, if you want help, you need to provide that. Not that this is a "version related issue".

In my experience, the nRF9160 runs with some SIM cards pretty stable (e.g. MTBF 100 days), and with others less (similar as your experience e.g. 30 days). The interesting point is, that my devices are using the same base stations. My current assumption is, that some components of the SIM provider don't report there issue back to the network and so the modem has no chance to get noticed. In the end, if I get for a certain amount of time no response from my server, I also reboot the device. Crossing the fingers, for now (unfortunately only 2-3 weeks for now), all outages had other explanations (e.g. removed batteries :-) ).