Help with LTE connection after loss of connection

[AnnaQ]

We have noticed that several of our nRF9160 in production have problem with reconnecting to the LTE-M network after temporarily losing connection. We have tried reproducing this issue by deactivating the sim for about 10 seconds and then activating it again. After the default six retries it gives up and stop trying to reconnect to the network. the only way we can achieve a connection is if we do a sys_reboot() or a hard reboot with watchdog. Why is the reconnect attempts not enough? Why do we have to force a reboot? Attached are relevant logs.

[Matias Marti]

[00:00:17.177,490] <inf> app_lwm2m_client: LwM2M is connected to server
[00:00:17.177,551] <inf> app_lwm2m_client: Obtained date-time from modem
[00:00:36.092,285] <dbg> app_lwm2m_client: rd_client_event: Registration update started
[00:01:16.382,965] <inf> net_lwm2m_rd_client: Update callback (code:2.4)
[00:01:16.383,026] <dbg> app_lwm2m_client: rd_client_event: Registration update complete
[00:01:16.383,056] <dbg> app_lwm2m_client: watchdog_Kick: Watchdog feed ok!
[00:01:16.383,148] <inf> net_lwm2m_rd_client: Update Done
[00:01:35.383,056] <dbg> app_lwm2m_client: rd_client_event: Registration update started
[00:02:15.828,491] <inf> net_lwm2m_rd_client: Update callback (code:2.4)
[00:02:15.828,552] <dbg> app_lwm2m_client: rd_client_event: Registration update complete
[00:02:15.828,582] <dbg> app_lwm2m_client: watchdog_Kick: Watchdog feed ok!
[00:02:15.828,643] <inf> net_lwm2m_rd_client: Update Done

It takes up to 40s for the registration update to complete when we set these config values:

CONFIG_COAP_RANDOMIZE_ACK_TIMEOUT=n
CONFIG_COAP_BACKOFF_PERCENT=100
CONFIG_COAP_MAX_RETRANSMIT=4

We are also getting the following logs:

[00:02:31.944,396] <dbg> app_lwm2m_client: watchdog_Kick: Watchdog feed ok!
[00:02:31.944,488] <inf> net_lwm2m_rd_client: Update Done
[00:02:32.430,450] <dbg> app_lwm2m_client: rd_client_event: Queue mode RX window closed
[00:02:50.944,335] <dbg> app_lwm2m_client: rd_client_event: Registration update started

When the server is slow to respond after an initial registration update, it seems like the queue mode RX gets closed right before the next registration update starts. Could this be some sort of race condition?

When our Leshan server responds, the logs just stay quiet until the watchdog kicks in.

[SeppoTakalo]

Then if the DTLS CID is not enabled in the server side, it might cause these connection issues.

When NAT timeout to happens, the IP&PORT pair mapping will change and it causes Leshan to ignore the LwM2M update messages because they come from unknown port. This offcourse only happens, if LwM2M engine is not instructed to do a DTLS Session resumption when doing a LwM2M Update.

My first recommendation would be to update Leshan, but with current environment, try:

# Do a DTLS Session resumption when coming
# back from RX_OFF state.
CONFIG_LWM2M_RD_CLIENT_SUSPEND_SOCKET_AT_IDLE=y
CONFIG_LWM2M_TLS_SESSION_CACHING=y

# Fine-tune the timeout to match NAT timeout on
# current network. I have seen 20s on some LTE networks in Finland.
# Try 10, 15, 20, 30
CONFIG_LWM2M_QUEUE_MODE_ENABLED=y
CONFIG_LWM2M_QUEUE_MODE_UPTIME=15

[Matias Marti]

 SeppoTakalo  thanks again.

Is there another way to prevent a NAT timeout? By setting  CONFIG_LWM2M_QUEUE_MODE_UPTIME=15 we are using all the layers of communication here below, right?

On which of these layers does the NAT timeout occur? Is there a way to use your SDK so that it sends a message to the LTE network on the lowest possible layer? We would like to prevent NAT timeout without communicating all the way to the Leshan server.

Any thoughts?

[SeppoTakalo]

https://en.wikipedia.org/wiki/Network_address_translation#One-to-many_NAT
NAT happens at IP layer.

So in order to keep the current mapping active, you must send UDP packets through in the higher frequency than what the network routers NAT timeout is.

So in a short: There is no way to prevent NAT timeout without sending packets.

If you want to prevent the timeout, you can configure for example CONFIG_LWM2M_UPDATE_PERIOD to be a small enough so that it does not cause NAT timeouts.

I would still recommend to keep the configs from previous example, so if for some reason, the timeout happens, next handshake would still be using session resumption.

There is a risk on that kind of configuration however: If we configure UPDATE_PERIOD shorter than QUEUE_MODE_UPTIME it means that LwM2M engine is never in so called RX_OFF state, or one might call it QUEUE mode. So it assumes that socket is constantly active, and does not try DTLS Session resumption.
If we end up sending LwM2M Update message into DTLS socket where NAT timeout have happened, it causes all DTLS packets to be ignore by the server. And then all the CoAP retry logic is just wasted time.

If we instead assume that NAT timeouts might happen, and allow those to happen, and configure UPDATE_PERIOD to a longer value, for example several minutes or hours, then LwM2M Update causes engine to "resume" the DTLS connection, which means that it actually closes the socket and does a new handshake using DTLS Session resumption. This is almost like full DTLS-handshake but shorter. It is accepted by the server because it starts with normal DTLS Client-Hello.

So any approach you take will consume a lot of bandwidth to keep the connection up.

With DTLS Connection-Identifier you get rid of the issues with NAT timeouts because server side uses CID to identify connections instead of IP&port pair. Then NAT re-mapping does not interrupt the DTLS session, it only block the server from communicating to the device, until it does LwM2M Update which refreshes the IP&port mapping.

[Achim Kraus]

Matias Marti said:

It seems like Leshan/Californium does not support DTLS CID. 

It's supported in Californium, I added the support pretty early during the development of RFC 9146 (I'm one of the co-authors).

Do you run Leshan on your own? Then you may need to enable CID via the configuration.

Unfortunately, it's not only the DTLS layer, which may get "mixed up" by the NAT changes. So you need also to consider Leshan's other settings to configure it proper (maybe you need to open an ticket/issue in the Leshan project.)