LwM2M client: registration SOMETIMES fails; observers not set.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Wed, 12 Jun 2024 15:26:00 GMT

I have made a new ticket describing behaviour that leads to crazy high power draw. I'd be thankful if you could have a look. I'll try to provide the modem trace tomorrow as I'll try to capture everything over night.

thanks :)

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Wed, 05 Jun 2024 09:32:36 GMT

Our MNO increased the APN's idle timeout and we no longer get the service reject. Also, testing with the Leshan M14 public server using composite observe we no longer face any issues.

As for now, I think we can consider this issue solved, so I will recap:

a.) The initial issue on toolchain v2.4.2 and mfw v1.3.4 was that the Server had no congestion control and just spammed the device with requests upon registration. The Device then at one point just closed its RX window, causing it to stop answering. Solutions are:

1. reducing requests from the server by configuring the server to send very few requests upon registration

2. Using a Server with congestion control (e.g. Leshan v2.0.0-M14)

3. Update Application to toolchain v2.6.0 or higher and mfw v1.3.6 or higher (paying attention to the compatibility matrix, of course) and use Seppo's recommended settings:

[quote userid="97896" url="~/f/nordic-q-a/110038/lwm2m-client-registration-sometimes-fails-observers-not-set/482503"]So for this traffic profile and servers, I recommend that you update clients to use DTLS CID and CONFIG_LWM2M_RD_CLIENT_LISTEN_AT_IDLE=y[/quote]

b.) CID is immensely helpful and reduces transmission times as well as improving functionality.

c.) After issue a. was solved, we had the issue that the device was unable to connect to the network after being in PSM for ~1h or longer and would perform a full reconnection and reregistration to the lwm2m server. This led to the discovery of unintended client behaviour, which Seppo created a PR for:

[quote userid="97896" url="~/f/nordic-q-a/110038/lwm2m-client-registration-sometimes-fails-observers-not-set/484626"]

Application should not react to registration timeout as it is something the engine itself should recover.

Can you try refactoring the main.c so that for events

LWM2M_RD_CLIENT_EVENT_REGISTRATION_FAILURE and LWM2M_RD_CLIENT_EVENT_REG_TIMEOUT

just call `k_mutex_unlock(&lte_mutex);` instead of this `state_trigger_and_unclock(NETWORK_ERROR)`

[/quote]

d.) After that, we discovered that our device was being detached from the network with the `Service reject (implicitly detached)` message in the modem trace. After contacting our Mobile Network Operator we have confirmed that the Access Point Name's (APN) idle timeout ignored the requested PSM timer and simply detached our device from the Network after 1h. They have now set the idle timeout to 24h which should allow for proper PSM behaviour. Also, our active devices with a 6h sending interval have stopped doing full reregistrations and are doing normal registration updates, which means that they no longer detach from the network.

e.) now testing everything with the Leshan M14 server, using toolchain version 2.6.0, mfw 1.3.6 and utilising the ObserveComposite function, everything is working as expected. Thingsboard will release their 3.7 Update with the updated LwM2M components (from Leshan v2.0.0-M5 to v2.0.0-M14) by the end of next week and we expect (and hope) that our application will run smoothly from then on out. Hopefully we can start using LwM2M as intended without all the workarounds and hacks we had to resort to until now.

Thank you so much Seppo and Achim for your valuable help and support, it was a pleasure learning from you.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

Achim Kraus — Tue, 04 Jun 2024 18:21:56 GMT

That's really great news.

With that new FW 2.0.1 it gets now possible to disable the HPPLMN search (if the use-case is indicating it) and to keep the DTLS context on network loss.

So only one of my "energy eater" tickets is not addressed. What's about the IDLE on failing network searches (for rescue calls). Is there also a plan to get rid of that?

RE: LwM2M client: registration SOMETIMES fails; observers not set.

SeppoTakalo — Tue, 04 Jun 2024 10:36:09 GMT

[quote userid="5203" url="~/f/nordic-q-a/110038/lwm2m-client-registration-sometimes-fails-observers-not-set/486588"]there is no reason for DTLS 1.2 CID to do a full handshake again. Your statement addresses only some Nordic implementation limits, and is not in general true. [/quote]

That is correct.

In nRF91 family devices, DTLS stack is inside modem and sockets are bind to PDN connections. So when network is lost, socket handles are closed, and so are DTLS sessions lost.

However, in modem firmware 2.0.1 there is a socket option SO_KEEPOPEN that will not close the socket while PDN is down. See https://docs.nordicsemi.com/bundle/ncs-latest/page/nrfxlib/nrf_modem/doc/sockets/socket_options_func.html#:~:text=modem%20firmware%20v2.0.0.-,NRF_SO_KEEPOPEN,-Keep%20the%20socket

When this is used with DTLS CID, it keeps the LwM2M session even when network is temporary lost.

But it requires different hardware as Modem FW 2.0.1 runs on nRF9161, nRF9151, nRF9131, which is not the case here.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

Achim Kraus — Thu, 30 May 2024 08:30:01 GMT

[quote userid="97896" url="~/f/nordic-q-a/110038/lwm2m-client-registration-sometimes-fails-observers-not-set/485796"]Their answer is that network is rejecting the re-attach after the sleep. There is nothing the modem can do, other than full handshake. But it looks like network induced issue.[/quote]

Just check my Zephyr coaps-client with tinydtls example and you will see, that even in cases of network loss and closed UDP sockets, there is no reason for DTLS 1.2 CID to do a full handshake again. Your statement addresses only some Nordic implementation limits, and is not in general true.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

SeppoTakalo — Wed, 29 May 2024 09:42:38 GMT

Try if you can request smaller TAU period. Preferably the same as your LwM2M update period.

Check following Kconfigs

LTE_PSM_REQ
LTE_PSM_REQ_FORMAT_SECONDS
LTE_PSM_REQ_RPTAU_SECONDS

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Wed, 29 May 2024 08:27:50 GMT

Small update:

I still haven't heard from our MNO, but I did do some research on LTE network procedures and the main suspect is the APN idle timer as described on page 12 of this document: GSMA LTE-M deployment guide, and on this page: Cisco idle timer. It seems like our device (the UE) is implicitly detached from the network within less than an hour of inactivity. This idle timer should be configurable in the Access Point Name (APN), which we will check with our MNO as soon as they find the time.

hope this helps someone in the future.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Fri, 24 May 2024 11:51:07 GMT

Thanks Seppo, I was just writing my next comment as your message arrived.

We took a step back and started testing with the unmodified LwM2M client sample. Using only the vanilla client sample of the v2.6.0 toolchain with mfw v1.3.6, we can regularly recreate the issue using a update time of 1 h and sometimes less.

using the overlay-lwm2m-1.1-core-interop.conf and the overlay-lwm2m-1.1.conf overlays as well as the following configs to connect to the open leshan server, the issue happened every time.

CONFIG_APP_LWM2M_PSK=""

CONFIG_LWM2M_CLIENT_UTILS_SERVER="coap://leshan.eclipseprojects.io"

CONFIG_LWM2M_PEER_PORT=5683

CONFIG_LWM2M_RD_CLIENT_SUPPORT_BOOTSTRAP=n

CONFIG_APP_ENDPOINT_PREFIX="urn:imei:"

CONFIG_LWM2M_DTLS_SUPPORT=y

  


CONFIG_LWM2M_DTLS_CID=y

CONFIG_LWM2M_CLIENT_UTILS_DTLS_CON_MANAGEMENT=n

CONFIG_LTE_LC_MODEM_SLEEP_NOTIFICATIONS=y

CONFIG_LWM2M_RD_CLIENT_LISTEN_AT_IDLE=n


CONFIG_LWM2M_UPDATE_PERIOD=3600

CONFIG_LWM2M_ENGINE_DEFAULT_LIFETIME=43260

CONFIG_LWM2M_SECONDS_TO_UPDATE_EARLY=60

  

CONFIG_LWM2M_LOG_LEVEL_DBG=y

  

CONFIG_THREAD_MONITOR=n

CONFIG_USE_SEGGER_RTT=y

CONFIG_STDOUT_CONSOLE=n

CONFIG_THREAD_NAME=n

CONFIG_NRF_MODEM_LIB_TRACE=n

Here you can see the corresponding device log:

[00:00:00.258,941] <err> SHT4X: Failed to reset the device.
*** Booting nRF Connect SDK v3.5.99-ncs1 ***
[00:00:00.261,260] <dbg> net_lwm2m_engine: lwm2m_engine_init: LWM2M engine socket receive thread started
[00:00:00.261,383] <dbg> net_lwm2m_obj_security: security_create: Create LWM2M security instance: 0
[00:00:00.261,535] <dbg> net_lwm2m_obj_server: server_create: Create LWM2M server instance: 0
[00:00:00.261,627] <dbg> net_lwm2m_obj_device: device_create: Create LWM2M device instance: 0
[00:00:00.261,962] <dbg> net_lwm2m_obj_conn_mon: connmon_create: Create LWM2M connectivity monitoring instance: 0
[00:00:00.262,023] <dbg> net_lwm2m_obj_firmware: firmware_create: Create LWM2M firmware instance: 0
[00:00:00.262,420] <inf> app_lwm2m_client: Run LWM2M client
[00:00:00.504,302] <inf> app_lwm2m_client: endpoint: urn:imei:350457793131176
[00:00:00.511,413] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:0/0/2, buf:0x20021337, len:1
[00:00:00.511,474] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:0/0/3, buf:0x2001b132, len:25
[00:00:00.619,323] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:0/0/10, buf:0x2002138e, len:2
[00:00:00.619,384] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:1/0/0, buf:0x2002138e, len:2
[00:00:00.656,005] <dbg> net_lwm2m_obj_portfolio: portfolio_create: Create LwM2M Portfolio instance: 0
[00:00:00.656,158] <inf> lwm2m_lte_notification: Registering ncell notification handler
[00:00:00.656,219] <inf> lwm2m_firmware: Image is confirmed OK
[00:00:00.656,250] <inf> app_lwm2m_client: Connecting to network.
[00:00:00.656,250] <inf> app_lwm2m_client: This may take several minutes.
[00:00:02.647,430] <dbg> app_lwm2m_client: lwm2m_lte_reg_handler_notify: LTE NW status: 2
[00:00:02.647,491] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/0, buf:0x20025aff, len:1
[00:00:04.201,141] <dbg> app_lwm2m_client: lwm2m_lte_reg_handler_notify: LTE NW status: 1
[00:00:04.202,148] <inf> app_lwm2m_client: Connected to LTE network
[00:00:04.202,301] <inf> app_lwm2m_client: Client connect to server
[00:00:04.202,362] <inf> net_lwm2m_rd_client: Start LWM2M Client: urn:imei:350457793131176
[00:00:04.202,880] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 1
[00:00:04.203,308] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 6
[00:00:04.203,369] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2001f358, buflen:4
[00:00:04.203,399] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/0/0, level 3, buf:0x2001f378, buflen:2
[00:00:04.203,460] <inf> net_lwm2m_rd_client: RD Client started with endpoint 'urn:imei:350457793131176' with client lifetime 43260 using server object 0
[00:00:04.203,521] <dbg> net_lwm2m_message_handling: lwm2m_parse_peerinfo: Parse url: coap://leshan.eclipseprojects.io
[00:00:04.204,711] <inf> app_lwm2m_client: LwM2M is connecting to server
[00:00:04.244,934] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/4, buf:0x20012c9a, len:13
[00:00:04.245,025] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/7, buf:0x20012f3a, len:14
[00:00:04.245,086] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/3, buf:0x20013172, len:18
[00:00:04.245,147] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/8, buf:0x20025b7c, len:4
[00:00:04.245,178] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/9, buf:0x20025b7e, len:2
[00:00:04.245,239] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/10, buf:0x20025b7e, len:2
[00:00:04.245,300] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/12, buf:0x20025b7e, len:2
[00:00:04.282,653] <inf> net_lwm2m_engine: Connected, sock id 2
[00:00:04.283,233] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2001f328, buflen:4
[00:00:04.283,691] <dbg> net_lwm2m_rd_client: sm_send_registration: registration sent [23.97.187.154]
[00:00:04.284,942] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 8
[00:00:04.293,304] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/2, buf:0x20025bd6, len:2
[00:00:04.434,417] <dbg> net_lwm2m_rd_client: do_registration_reply_cb: Registration callback (code:2.1)
[00:00:04.435,272] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2001f190, buflen:4
[00:00:04.435,882] <dbg> app_lwm2m_client: rd_client_event: Registration complete
[00:00:04.436,035] <inf> net_lwm2m_rd_client: Registration Done (EP='I9vE74GufA')
[00:00:04.436,187] <dbg> net_lwm2m_message_handling: lwm2m_udp_receive: reply 0x2000d1d0 handled and removed
[00:00:04.436,248] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 9
[00:00:04.436,889] <inf> app_lwm2m_client: LwM2M is connected to server
[00:00:04.436,920] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x200213f8, buflen:4
[00:00:04.437,042] <inf> app_lwm2m_client: Obtained date-time from modem
[00:00:04.437,103] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/13, buf:0x2001fa9c, len:4
[00:00:34.284,973] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 9
[00:00:34.285,034] <inf> lwm2m_rai: RAI enabled
[00:00:34.285,034] <dbg> app_lwm2m_client: rd_client_event: Queue mode RX window closed
[00:01:01.033,020] <inf> app_lwm2m_client: Modem Enter PSM, time 46769950
[01:00:04.434,570] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 10
[01:00:04.434,661] <dbg> app_lwm2m_client: rd_client_event: Registration update started
[01:00:04.535,064] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 11
[01:00:04.535,095] <dbg> net_lwm2m_engine: lwm2m_engine_connection_resume: Resume suspended connection
[01:00:04.535,644] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2001f330, buflen:4
[01:00:04.535,797] <dbg> net_lwm2m_rd_client: sm_send_registration: registration sent [23.97.187.154]
[01:00:04.535,919] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 12
[01:00:04.625,549] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/2, buf:0x20025bd6, len:2
[01:00:06.455,169] <err> net_lwm2m_engine: Poll reported a socket error, 08.
[01:00:06.455,200] <err> net_lwm2m_rd_client: RD Client socket error: 5
[01:00:06.455,596] <inf> lwm2m_rai: RAI disabled
[01:00:06.455,596] <dbg> app_lwm2m_client: rd_client_event: Registration update failure!
[01:00:06.455,657] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 18
[01:00:06.455,688] <err> net_lwm2m_rd_client: sm_do_network_error, retries 0
[01:00:06.456,329] <err> net_lwm2m_engine: Cannot connect UDP (-114)
[01:00:06.456,604] <err> net_lwm2m_rd_client: Failed to start socket -114
[01:00:06.457,122] <inf> net_lwm2m_rd_client: Stop LWM2M Client: urn:imei:350457793131176
[01:00:06.457,183] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 17
[01:00:06.457,672] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 0
[01:00:06.459,228] <inf> app_lwm2m_client: Client connect to server
[01:00:06.459,259] <inf> net_lwm2m_rd_client: Start LWM2M Client: urn:imei:350457793131176
[01:00:06.459,381] <dbg> app_lwm2m_client: lwm2m_lte_reg_handler_notify: LTE NW status: 2
[01:00:06.459,625] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 1
[01:00:06.459,838] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 6
[01:00:06.459,869] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2001f358, buflen:4
[01:00:06.459,930] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/0/0, level 3, buf:0x2001f378, buflen:2
[01:00:06.459,991] <inf> net_lwm2m_rd_client: RD Client started with endpoint 'urn:imei:350457793131176' with client lifetime 43260 using server object 0
[01:00:06.460,052] <dbg> net_lwm2m_message_handling: lwm2m_parse_peerinfo: Parse url: coap://leshan.eclipseprojects.io
[01:00:06.460,327] <inf> app_lwm2m_client: LwM2M is connecting to server
[01:00:06.460,479] <err> net_lwm2m_message_handling: Unable to resolve address
[01:00:06.460,479] <err> net_lwm2m_rd_client: Cannot init LWM2M engine (-2)
[01:00:06.460,845] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 18
[01:00:06.460,845] <err> net_lwm2m_rd_client: sm_do_network_error, retries 0
[01:00:07.460,815] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 18
[01:00:07.460,876] <err> net_lwm2m_rd_client: sm_do_network_error, retries 0
[01:00:07.461,029] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 6
[01:00:07.461,059] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2001f358, buflen:4
[01:00:07.461,120] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/0/0, level 3, buf:0x2001f378, buflen:2
[01:00:07.461,181] <inf> net_lwm2m_rd_client: RD Client started with endpoint 'urn:imei:350457793131176' with client lifetime 43260 using server object 0
[01:00:07.461,242] <dbg> net_lwm2m_message_handling: lwm2m_parse_peerinfo: Parse url: coap://leshan.eclipseprojects.io
[01:00:07.461,547] <err> net_lwm2m_message_handling: Unable to resolve address
[01:00:07.461,547] <err> net_lwm2m_rd_client: Cannot init LWM2M engine (-2)
[01:00:07.461,914] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 18
[01:00:07.461,944] <err> net_lwm2m_rd_client: sm_do_network_error, retries 1
[01:00:08.462,890] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 18
[01:00:08.462,951] <err> net_lwm2m_rd_client: sm_do_network_error, retries 1
[01:00:08.463,256] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 6
[01:00:08.463,287] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2001f358, buflen:4
[01:00:08.463,317] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/0/0, level 3, buf:0x2001f378, buflen:2
[01:00:08.463,378] <inf> net_lwm2m_rd_client: RD Client started with endpoint 'urn:imei:350457793131176' with client lifetime 43260 using server object 0
[01:00:08.463,439] <dbg> net_lwm2m_message_handling: lwm2m_parse_peerinfo: Parse url: coap://leshan.eclipseprojects.io
[01:00:08.463,745] <err> net_lwm2m_message_handling: Unable to resolve address
[01:00:08.463,745] <err> net_lwm2m_rd_client: Cannot init LWM2M engine (-2)
[01:00:08.463,958] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 18
[01:00:08.464,019] <err> net_lwm2m_rd_client: sm_do_network_error, retries 2
[01:00:08.520,477] <dbg> app_lwm2m_client: lwm2m_lte_reg_handler_notify: LTE NW status: 1
[01:00:08.520,660] <inf> app_lwm2m_client: LwM2M is connecting to server
[01:00:08.563,659] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/4, buf:0x20012c9a, len:13
[01:00:08.563,720] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/7, buf:0x20012f3a, len:14
[01:00:08.563,781] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/3, buf:0x20013172, len:18
[01:00:08.563,812] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/8, buf:0x20025b7c, len:4
[01:00:08.563,873] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/9, buf:0x20025b7e, len:2
[01:00:08.563,903] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/10, buf:0x20025b7e, len:2
[01:00:08.563,934] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/12, buf:0x20025b7e, len:2
[01:00:08.564,117] <inf> app_lwm2m_client: Obtained date-time from modem
[01:00:08.564,178] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/13, buf:0x20025b14, len:4
[01:00:10.464,965] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 18
[01:00:10.465,026] <err> net_lwm2m_rd_client: sm_do_network_error, retries 2
[01:00:10.465,332] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 6
[01:00:10.465,362] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2001f358, buflen:4
[01:00:10.465,393] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/0/0, level 3, buf:0x2001f378, buflen:2
[01:00:10.465,454] <inf> net_lwm2m_rd_client: RD Client started with endpoint 'urn:imei:350457793131176' with client lifetime 43260 using server object 0
[01:00:10.465,515] <dbg> net_lwm2m_message_handling: lwm2m_parse_peerinfo: Parse url: coap://leshan.eclipseprojects.io
[01:00:10.466,613] <inf> net_lwm2m_engine: Connected, sock id 2
[01:00:10.467,163] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2001f328, buflen:4
[01:00:10.467,620] <dbg> net_lwm2m_rd_client: sm_send_registration: registration sent [23.97.187.154]
[01:00:10.468,780] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 8
[01:00:12.172,027] <dbg> net_lwm2m_rd_client: do_registration_reply_cb: Registration callback (code:2.1)
[01:00:12.173,004] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2001f190, buflen:4
[01:00:12.174,285] <dbg> app_lwm2m_client: rd_client_event: Registration complete
[01:00:12.174,438] <inf> net_lwm2m_rd_client: Registration Done (EP='qtLEAQMa6z')
[01:00:12.174,591] <dbg> net_lwm2m_message_handling: lwm2m_udp_receive: reply 0x2000d1d0 handled and removed
[01:00:12.174,652] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 9
[01:00:12.175,262] <inf> app_lwm2m_client: LwM2M is connected to server
[01:00:12.175,323] <inf> app_lwm2m_client: Obtained date-time from modem
[01:00:12.175,384] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/13, buf:0x2001fa9c, len:4
[01:00:40.468,353] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 9
[01:00:40.468,414] <inf> lwm2m_rai: RAI enabled
[01:00:40.468,414] <dbg> app_lwm2m_client: rd_client_event: Queue mode RX window closed
[01:01:08.709,533] <inf> app_lwm2m_client: Modem Enter PSM, time 50369950

If someone would be willing to test this with a thingy91 in a different country, we would be grateful, although it is really starting to look like it might be specific to our local Network Provider or something like that.

We will now start experimenting some more with PSM (haven't really been using eDRX) and will contact our network provider.

Thanks for investigating some more 🙂

RE: LwM2M client: registration SOMETIMES fails; observers not set.

SeppoTakalo — Fri, 24 May 2024 11:33:54 GMT

I tried to ask modem guys about this highlighted part.

Their answer is that network is rejecting the re-attach after the sleep. There is nothing the modem can do, other than full handshake. But it looks like network induced issue.

This is not really anymore my expertise by I would try to evaluate different PSM and eDRX configurations if you can find a combination that does not cause network to drop the modem.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Wed, 22 May 2024 12:42:39 GMT

Thank you Seppo

We will start investigating network timeouts and resume testing with other servers then. I will let you know as soon as we learn the cause of the error.

Here's the corresponding modem trace anyway

devzone.nordicsemi.com/.../202405221431_5F00_2h_5F00_probable_5F00_socket_5F00_error_5F00_recovery_5F00_wo_5F00_decryption.pcapng

RE: LwM2M client: registration SOMETIMES fails; observers not set.

SeppoTakalo — Wed, 22 May 2024 12:01:58 GMT

net_lwm2m_engine: Poll reported a socket error, 08.
app_lwm2m_client: lwm2m_lte_reg_handler_notify: LTE NW status: 2
net_lwm2m_engine: Cannot connect UDP (-114)

These all are indications that modem has lost the network connection.

Usually then network is lost, poll() returns POLLERR for the socket. This is because sockets are tied to specific PDN on the modem, so when PDN is lost, socket is lost too.

Same indication is from the lte_reg_handler

	/**
	 * Not registered, but UE is currently trying to attach or searching for an operator to
	 * register to.
	 */
	LTE_LC_NW_REG_SEARCHING			= 2,

So the network is lost and modem is searching the network.

The UDP erro -114 is ENETUNREACH from the same reason, network is lost.

So try to find the reason why the LTE connection is lost. This does not seem LwM2M issue anymore.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Wed, 22 May 2024 10:16:07 GMT

Hi Seppo

So we implemented the State Machine change (Unlock instead of trigger Network_error) and the device no longer performs full reregistrations. It does recover from the UDP -114 Error without a full reregistration, however, the Error seems to happen every time.

Until I can provide the Modemtrace, here's the device log:

[02:00:09.449,005] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3316/0/5700, buf:0x2002f5e0, len:8
Battery voltage is 2958 mV

[02:00:11.433,258] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/7, buf:0x2002f614, len:4
[02:00:16.004,730] <dbg> net_lwm2m_message_handling: generate_notify_message: [AUTO] NOTIFY MSG START: 3316/0/5700(3) token:'fd5aea5ae5863ebe' [34.65.106.240] 7216004
[02:00:16.006,530] <dbg> net_lwm2m_message_handling: generate_notify_message: NOTIFY MSG: SENT
[02:00:16.006,713] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 9
[02:00:16.019,042] <dbg> app_lwm2m_client: rd_client_event: Registration update started
[02:00:16.119,598] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 11
[02:00:16.119,628] <dbg> net_lwm2m_engine: lwm2m_engine_connection_resume: Resume suspended connection
[02:00:16.120,208] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x20028d78, buflen:4
[02:00:16.120,361] <dbg> net_lwm2m_rd_client: sm_send_registration: registration sent [34.65.106.240]
[02:00:16.129,394] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 12
[02:00:16.209,716] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/2, buf:0x2002f61e, len:2
[02:00:17.674,316] <err> net_lwm2m_engine: Poll reported a socket error, 08.
[02:00:17.674,346] <err> net_lwm2m_rd_client: RD Client socket error: 5
[02:00:17.675,140] <inf> lwm2m_rai: RAI disabled
[02:00:17.675,170] <dbg> app_lwm2m_client: rd_client_event: Registration update failure!
[02:00:17.675,201] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 18
[02:00:17.675,231] <err> net_lwm2m_rd_client: sm_do_network_error, retries 0
[02:00:17.675,323] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:0/0/2/0, level 3, buf:0x20028d7b, buflen:1
[02:00:17.677,856] <dbg> app_lwm2m_client: lwm2m_lte_reg_handler_notify: LTE NW status: 2
[02:00:17.758,361] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:0/0/2/0, level 3, buf:0x20028d43, buflen:1
[02:00:17.759,002] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:0/0/2/0, level 3, buf:0x20028d7b, buflen:1
[02:00:17.759,643] <err> net_lwm2m_engine: Cannot connect UDP (-114)
[02:00:17.759,887] <err> net_lwm2m_rd_client: Failed to start socket -114
[02:00:17.760,131] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_pause: lwm2m_rd_client_pause()
[02:00:17.760,162] <inf> net_lwm2m_rd_client: Suspend client
[02:00:17.760,192] <inf> lwm2m_rai: RAI enabled
[02:00:17.760,192] <dbg> app_lwm2m_client: rd_client_event: LwM2M engine suspended
[02:00:17.769,439] <inf> net_lwm2m_engine: LWM2M engine thread paused
[02:00:17.769,470] <inf> app_lwm2m_client: LTE Offline
[02:00:19.589,782] <dbg> app_lwm2m_client: lwm2m_lte_reg_handler_notify: LTE NW status: 1
[02:00:19.589,996] <inf> app_lwm2m_client: Resume LwM2M engine
[02:00:19.590,118] <inf> net_lwm2m_rd_client: Resume Client state
[02:00:19.590,423] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 18
[02:00:19.590,423] <err> net_lwm2m_rd_client: sm_do_network_error, retries 1
[02:00:19.593,017] <inf> app_lwm2m_client: LwM2M is connecting to server
[02:00:19.632,507] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/4, buf:0x20015af2, len:13
[02:00:19.632,568] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/7, buf:0x20015d92, len:14
[02:00:19.632,629] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/3, buf:0x20015fca, len:18
[02:00:19.632,690] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/8, buf:0x2002f5c4, len:4
[02:00:19.632,751] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/9, buf:0x2002f5c6, len:2
[02:00:19.632,781] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/10, buf:0x2002f5c6, len:2
[02:00:19.632,812] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:4/0/12, buf:0x2002f5c6, len:2
[02:00:19.632,995] <inf> app_lwm2m_client: Obtained date-time from modem
[02:00:19.633,026] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/13, buf:0x2002f55c, len:4
[02:00:20.590,728] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 18
[02:00:20.590,789] <err> net_lwm2m_rd_client: sm_do_network_error, retries 1
[02:00:20.590,881] <inf> lwm2m_rai: RAI disabled
[02:00:20.590,881] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:0/0/2/0, level 3, buf:0x20028d7b, buflen:1
[02:00:20.660,308] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:0/0/2/0, level 3, buf:0x20028d43, buflen:1
[02:00:20.661,071] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:0/0/2/0, level 3, buf:0x20028d7b, buflen:1
[02:00:20.949,798] <inf> net_lwm2m_engine: Connected, sock id 2
[02:00:20.949,829] <inf> lwm2m_rai: RAI enabled
[02:00:20.949,859] <dbg> app_lwm2m_client: rd_client_event: Registration update started
[02:00:20.950,164] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 11
[02:00:20.950,683] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x20028d78, buflen:4
[02:00:20.950,836] <dbg> net_lwm2m_rd_client: sm_send_registration: registration sent [34.65.106.240]
[02:00:20.950,958] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 12
[02:00:21.094,604] <inf> net_lwm2m_rd_client: Update callback (code:2.4)
[02:00:21.094,665] <dbg> app_lwm2m_client: rd_client_event: Registration update complete
[02:00:21.094,787] <inf> net_lwm2m_rd_client: Update Done
[02:00:21.095,001] <dbg> net_lwm2m_message_handling: lwm2m_udp_receive: reply 0x2000e268 handled and removed
[02:00:21.095,062] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 9
[02:00:21.095,855] <inf> app_lwm2m_client: LwM2M is connected to server
[02:00:21.096,038] <inf> app_lwm2m_client: Obtained date-time from modem
[02:00:21.096,069] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/13, buf:0x200294e4, len:4
[02:00:21.253,326] <dbg> net_lwm2m_message_handling: notify_message_reply_cb: NOTIFY ACK type:2 code:0.0 reply_token:'fd5aea5ae5863ebe'
[02:00:21.253,540] <dbg> net_lwm2m_message_handling: lwm2m_udp_receive: reply 0x2000e250 handled and removed
[02:00:23.953,002] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 9
[02:00:24.096,527] <dbg> net_lwm2m_rd_client: lwm2m_rd_client_service: State: 9
[02:00:24.105,651] <dbg> app_lwm2m_client: rd_client_event: Queue mode RX window closed
[02:00:39.437,774] <inf> app_lwm2m_client: Modem Enter PSM, time 50383950
[02:01:05.362,670] <dbg> app_lwm2m_client: debug_connection_work_cb: current client_state: 3
[02:01:05.362,701] <dbg> app_lwm2m_client: debug_connection_work_cb: current retry_state: 100
[02:01:05.362,701] <dbg> app_lwm2m_client: debug_connection_work_cb: last lwm2m_rd_client_event: 10
[02:01:05.362,731] <dbg> app_lwm2m_client: debug_connection_work_cb: last lte_lc_nw_reg_status: 1
[02:01:05.362,762] <dbg> app_lwm2m_client: debug_connection_work_cb: modem_connected_to_network: 1
[02:01:05.362,762] <dbg> app_lwm2m_client: debug_connection_work_cb: current_uptime: 7265362
[02:01:05.362,792] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_reg_complete: 15711
[02:01:05.362,823] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_reg_timeout: 7217675
[02:01:05.362,823] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_reg_update: 7220949
[02:01:05.362,854] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_reg_update_complete: 7221094
[02:01:05.362,884] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_rx_off: 7224105
[02:01:05.362,915] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_engine_suspended: 7217760

By the way: I stumbled upon the OMA LwM2M Object 22 (Virtual Observe Notify Object), which would fit our needs. However, it is not implemented in the zephyr client, right? do you know if there is some implementation somewhere or has it become deprecated with the ObserveComposite operation?

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Tue, 21 May 2024 08:29:00 GMT

Yeah, my mistake, had a mixup with the firmware, devices and the 2 different servers... as you say, the credentials were not correctly defined on the server.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

SeppoTakalo — Tue, 21 May 2024 08:08:48 GMT

[quote userid="131108" url="~/f/nordic-q-a/110038/lwm2m-client-registration-sometimes-fails-observers-not-set/484985"]I managed to capture the observer update with wireshark and the joulescope.[/quote]

I don't think this is a correct capture.

What I see there in the Wireshark is just two failed DTLS handshakes and some HTTPS traffic from 172.217.168.42 which the device is trying to reply with RST.

So clearly the device is not registered.

I don't know where the TLS/HTTPS traffic is coming from.. has there been FOTA download ongoing and it has not been closed succesfully, or is this some weird form of port scanning.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Tue, 21 May 2024 07:39:39 GMT

Hi Achim

Thanks for the Input.

I managed to capture the observer update with wireshark and the joulescope.

here's the wireshark capture of the modem trace (Observer update from lines 191 to 204):

devzone.nordicsemi.com/.../202405210923_5F00_CID_5F00_key_5F00_exchange_5F00_retries_5F00_with_5F00_observer_5F00_update.pcapng

And the joulescope output (Observer update in red):

edit: i think it might be user error, give me a second to investigate. maybe just wrong credentials on platform.

edit2: yup, my bad. I'll leave it up in case it helps anyone else.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

Achim Kraus — Fri, 17 May 2024 14:02:26 GMT

The capture shows the retransmission of the last client's flight. For PSK that indicates mostely, that the "pre shared secret" is different. Please check that.

Sometime it's even that simple, that one side takes a "hex-string" as text, while the other decode it to binary.

e.g.

"abcd" is either

two bytes 0xab 0xcd or

four bytes 0x61 0x62 0x63 0x64

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Fri, 17 May 2024 13:52:43 GMT

So we've managed to observe a device just doing a normal registration instead of a full reregistration after 2 hours, which is great.

On the other hand, we've just started to get a weird registration behaviour on multiple of our devices that might be linked to CID.

Here's the trace file:

devzone.nordicsemi.com/.../202405171546_5F00_CID_5F00_registration_5F00_problems.pcapng

And the corresponding current measurement:

Seppo or Achim, are you familiar with what is going on?

Thanks for your continued support.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Thu, 16 May 2024 15:29:37 GMT

devzone.nordicsemi.com/.../202405161621_5F00_testserver_5F00_v2.6.0.pcapng

Thanks Seppo

I have made a tracefile, but I haven't had the time to capture a reregistration after the 2h. I hope to get around to that tomorrow.

Maybe you can find something in this file, else the file tomorrow should be more interesting.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

SeppoTakalo — Thu, 16 May 2024 10:07:30 GMT

I created a PR for these events

https://github.com/nrfconnect/sdk-nrf/pull/15338

It is enough that client restarts the LTE when LwM2M engine runs out of retries

[00:20:16.631,77] <er> net_lwm2m_engine: Cannot connect UDP (-116)
[00:20:16.632,751] <err> net_lwm2m_rd_client: Cannot init LWM2M engine (-116)
[00:20:16.633,087] <err> net_lwm2m_rd_client: sm_do_network_error, retries 5
uart:~$ [00:20:32.633,392] <err> net_lwm2m_rd_client: sm_do_network_error, retries 5
[00:20:32.633,483] <err> net_lwm2m_rd_client: Network error, max retries reached (6)
[00:20:32.633,514] <err> net_lwm2m_obj_server: No server candidate found
[00:20:32.633,544] <err> app_lwm2m_client: LwM2M engine reported a network error.
[00:20:32.634,307] <inf> net_lwm2m_rd_client: Stop LWM2M Client: urn:imei:351358814369747
[00:20:32.634,887] <inf> app_lwm2m_client: LwM2M restart requested. The sample will try to re-establish network connection.
[00:20:33.358,459] <dbg> app_lwm2m_client: lwm2m_lte_reg_handler_notify: LTE NW status: 0
[00:20:34.744,628] <inf> app_lwm2m_client: Connecting to network.
[00:20:34.744,659] <inf> app_lwm2m_client: This may take several minutes.
[00:20:35.888,549] <dbg> app_lwm2m_client: lwm2m_lte_reg_handler_notify: LTE NW status: 2
[00:20:36.616,058] <dbg> app_lwm2m_client: lwm2m_lt_hndler_notify: LTE NW status: 1
[00:20:36.617,126] <inf> app_lwm2m_client: Connected to LTE network
[00:20:36.617,248] <inf> app_lwm2m_client: Client connect to server
[00:20:36.617,309] <inf> net_lwm2m_rd_client: Start LWM2M Client: urn:imei:351358814369747
[00:20:36.618,408] <inf> net_lwm2m_rd_client: RD Client started with endpoint 'urn:imei:351358814369747' with client lifetime 43200 using server0
[00:20:36.619,171] <inf> app_lwm2m_client: LwM2M is connecting to server

RE: LwM2M client: registration SOMETIMES fails; observers not set.

SeppoTakalo — Thu, 16 May 2024 08:17:13 GMT

[quote userid="131108" url="~/f/nordic-q-a/110038/lwm2m-client-registration-sometimes-fails-observers-not-set/484612"]What's happening, is that the client times out trying to do the registration update, which triggers the NETWORK_ERROR state:[/quote]

I think this is a bug in the nRF SDK's LwM2M sample.

I believe it is not accurately following the state machine described here: https://docs.zephyrproject.org/latest/connectivity/networking/api/lwm2m.html#lwm2m-engine-and-application-events

Application should not react to registration timeout as it is something the engine itself should recover.

Can you try refactoring the main.c so that for events

LWM2M_RD_CLIENT_EVENT_REGISTRATION_FAILURE and LWM2M_RD_CLIENT_EVENT_REG_TIMEOUT

just call `k_mutex_unlock(&lte_mutex);` instead of this `state_trigger_and_unclock(NETWORK_ERROR)`

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Thu, 16 May 2024 06:32:43 GMT

Hi Seppo

I let the new application run overnight with a 2h CONFIG_LWM2M_UPDATE_PERIOD with no observers set, and we got full registrations each time.

[quote userid="97896" url="~/f/nordic-q-a/110038/lwm2m-client-registration-sometimes-fails-observers-not-set/483461"]There is also `net_lwm2m_rd_client: Suspend client`. What is causing those? Current client at lest does suspending when the LTE connection is lost. If this the case? Poor network/antenna? Or different application?[/quote]

What's happening, is that the client times out trying to do the registration update, which triggers the NETWORK_ERROR state:

	case LWM2M_RD_CLIENT_EVENT_REG_TIMEOUT:
		#if defined(CONFIG_PRIOT_CONNECTION_DEBUGGING)
			ts_last_reg_timeout = k_uptime_get(); 
			last_rd_event = LWM2M_RD_CLIENT_EVENT_REG_TIMEOUT;
		#endif
		LOG_DBG("Registration update failure!");
		state_trigger_and_unlock(NETWORK_ERROR);
		break;

Which in turn stops the rd client with lwm2m_rd_client_stop():

		case NETWORK_ERROR:
			/* Stop the LwM2M engine. */
			state_trigger_and_unlock(START);
			lwm2m_rd_client_stop(&client, rd_client_event, false);
			
			etc.

This timeout happens after ~1.6 seconds. You can see this in the device logs that record the timestamps (k_uptime_get() in ms) of the last rd_client_events:

[15:00:05.362,670] <dbg> app_lwm2m_client: debug_connection_work_cb: current client_state: 3
[15:00:05.362,731] <dbg> app_lwm2m_client: debug_connection_work_cb: current retry_state: 100
[15:00:05.362,731] <dbg> app_lwm2m_client: debug_connection_work_cb: last lwm2m_rd_client_event: 10
[15:00:05.362,762] <dbg> app_lwm2m_client: debug_connection_work_cb: last lte_lc_nw_reg_status: 1
[15:00:05.362,792] <dbg> app_lwm2m_client: debug_connection_work_cb: modem_connected_to_network: 1
[15:00:05.362,792] <dbg> app_lwm2m_client: debug_connection_work_cb: current_uptime: 54005362
[15:00:05.362,823] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_reg_complete: 50451607
[15:00:05.362,854] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_reg_timeout: 50447829
[15:00:05.362,854] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_reg_update: 50446183
[15:00:05.362,884] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_rx_off: 50455087

So what's happening is definitely that the registration update times out, triggers the network_error state which restarts the client and modem and triggers a full reregistration.

Question is now, why that timeout occurs and if it's too short.

Would a modem trace help?

I will also test what happens if the device connects after 2h with an observer set, instead of triggering the update with UPDATE_PERIOD.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Wed, 15 May 2024 12:50:21 GMT

[quote userid="97896" url="~/f/nordic-q-a/110038/lwm2m-client-registration-sometimes-fails-observers-not-set/483461"]DTLS CID should help this, because just LwM2M Update should refresh the NAT mappings and allow end to end connection.[/quote]

I'm testing this in the next couple of days and will let you know if any problems occur with the updated application.

I'll also look into the `net_lwm2m_rd_client: Suspend client` message. Same application, with the same modifications as before. I'll have a look later.

Thanks for the insight on the LwM2M engine and the confirmation 🙂

RE: LwM2M client: registration SOMETIMES fails; observers not set.

SeppoTakalo — Wed, 15 May 2024 10:48:07 GMT

[quote userid="131108" url="~/f/nordic-q-a/110038/lwm2m-client-registration-sometimes-fails-observers-not-set/483453"]With the previous application we also had the error -115 on registration updates, where the device kept falling back to a full re-registration[/quote]

-115 is

#define ENETDOWN 115     /**< Network is down */

I'm now unsure if this is caused by actual network problem, or just somehow error coming from DTLS stack when it times out during the handshake.

The logic on client side is that when CoAP retry does not recover the situation, it needs to tear down the DTLS session and start from full handshake. If the issue was just NAT timeout, then this resolves the situation. DTLS CID should help this, because just LwM2M Update should refresh the NAT mappings and allow end to end connection.

There is also `net_lwm2m_rd_client: Suspend client`. What is causing those? Current client at lest does suspending when the LTE connection is lost. If this the case? Poor network/antenna? Or different application?

[quote userid="131108" url="~/f/nordic-q-a/110038/lwm2m-client-registration-sometimes-fails-observers-not-set/483453"]Now, observing all the resources individually, they start to diverge after a day or week or month and instead of the ~7 observed values being sent at once, the device sends one value, ends transmission, drops connection and then immediately starts a new connection to send the next value. [/quote]

I see the problem.

Composite observation would be the answer.

This is really an issue on how the Zephyr's LwM2M engine is designed. Each observation have their individual timestamps. Technically it is correct, but for better energy use the timestamp accuracy should be lower, meaning there is some granularity, so that timestamps with same pMax would trigger on same timestamp, instead of 1 second apart. I think this should be reported to Zephyr side. Someone might be able to work on this in the future.

Regarding the Observer-Composite, at least the current Leshan should support it. I have already tested this

https://github.com/zephyrproject-rtos/zephyr/blob/main/tests/net/lib/lwm2m/interop/README.md

LightweightM2M-1.1-int-304 - Observe-Composite Operation	✅
LightweightM2M-1.1-int-305 - Cancel Observation-Composite Operation	✅
LightweightM2M-1.1-int-308 - Observe-Composite and Creating Object Instance	✅	~~#64634~~
LightweightM2M-1.1-int-309 - Observe-Composite and Deleting Object Instance	✅	~~#64634~~
LightweightM2M-1.1-int-310 - Observe-Composite and modification of parameter values

If your server does not support it, then I would recommend to use LwM2M SEND and have the triggering logic on device side. It is true, that it is less flexible, but it still straight forward to implement, and leads to best energy efficiency. There is also a hack (can be enabled by Kconfig) that allows skipping the LwM2M Update when sending data. Some servers seem to support this, at least when DTLS CID is used.

config LWM2M_QUEUE_MODE_NO_MSG_BUFFERING
	bool "Disable buffering notifications and messages on queue mode"
	select EXPERIMENTAL
	help
	  Messages are sent right away instead of waiting for next registration update.
	  This might not be supported on all servers.

RE: LwM2M client: registration SOMETIMES fails; observers not set.

AlanIoT — Wed, 15 May 2024 09:50:49 GMT

Thank you for looking further into the issue, Seppo

I have finally had the time to update the application, which is now running on MFW v1.3.6 and SDK v2.6.0.

CID is looking good, energy consumption is down by quite a bit. I have also not encountered the Issue above, but I have also not really been able to test the issue specifically.

With the previous application we also had the error -115 on registration updates, where the device kept falling back to a full re-registration. It happened very often for devices with a registration update time of 2, 3, 4, 5 and 6 hours (did not test lower intervals). I hope this will be fixed with the new client with CID and will keep you updated. See the following device log of a device with a 6h registration update time:

[06:00:35.228,607] <inf> app_lwm2m_client: Obtained date-time from modem
[06:00:35.228,668] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/13, buf:0x2003184c, len:4
[06:00:35.795,104] <err> net_lwm2m_engine: Cannot connect UDP (-115)
[06:00:35.796,447] <err> net_lwm2m_rd_client: Failed to update registration. Falling back to full registration
[06:00:36.296,691] <inf> net_lwm2m_rd_client: Suspend client
[06:00:36.296,722] <dbg> app_lwm2m_client: rd_client_event: LwM2M engine suspended
[06:00:36.306,579] <inf> net_lwm2m_engine: LWM2M engine thread paused
[06:00:36.306,610] <inf> app_lwm2m_client: Resume LwM2M engine
[06:00:36.306,671] <inf> net_lwm2m_rd_client: Resume Client state
[06:00:36.306,762] <inf> net_lwm2m_engine: LWM2M engine thread resume
[06:00:36.306,793] <inf> app_lwm2m_client: LwM2M is connecting to server
[06:00:36.307,067] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:0/0/1/0, level 3, buf:0x2002b167, buflen:1
[06:00:36.307,128] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:0/0/10/0, level 3, buf:0x2002b196, buflen:2
[06:00:36.307,159] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2002b150, buflen:4
[06:00:36.307,250] <inf> net_lwm2m_rd_client: RD Client started with endpoint 'urn:imei:350457793131259' with client lifetime 25200
[06:00:36.307,312] <dbg> net_lwm2m_message_handling: lwm2m_parse_peerinfo: Parse url: coaps://prod.lwm2m.measuremind.ch
[06:00:36.397,644] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:0/0/2/0, level 3, buf:0x2002b0e3, buflen:1
[06:00:36.479,431] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:0/0/2/0, level 3, buf:0x2002b0ab, buflen:1
[06:00:36.898,773] <inf> net_lwm2m_engine: Connected, sock id 0
[06:00:36.899,444] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2002b100, buflen:4
[06:00:36.900,756] <dbg> net_lwm2m_rd_client: sm_send_registration: registration sent [34.65.106.240]
[06:00:37.099,670] <dbg> net_lwm2m_rd_client: do_registration_reply_cb: Registration callback (code:2.1)
[06:00:37.099,731] <dbg> app_lwm2m_client: rd_client_event: Registration complete
[06:00:37.099,792] <inf> net_lwm2m_rd_client: Registration Done (EP='ahL2AsEXDu')
[06:00:37.099,975] <dbg> net_lwm2m_message_handling: lwm2m_udp_receive: reply 0x20015674 handled and removed
[06:00:37.100,128] <inf> app_lwm2m_client: LwM2M is connected to server
[06:00:37.100,189] <inf> app_lwm2m_client: Obtained date-time from modem
[06:00:37.100,250] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3/0/13, buf:0x2002b7dc, len:4
[06:00:37.402,221] <dbg> app_lwm2m_client: rd_client_event: Registration update started
[06:00:37.903,015] <dbg> net_lwm2m_registry: lwm2m_engine_get: path:1/0/1/0, level 3, buf:0x2002b110, buflen:4
[06:00:37.903,106] <dbg> net_lwm2m_rd_client: sm_send_registration: registration sent [34.65.106.240]
[06:00:38.160,186] <inf> net_lwm2m_rd_client: Update callback (code:2.4)
[06:00:38.160,247] <dbg> app_lwm2m_client: rd_client_event: Registration update complete
[06:00:38.160,278] <inf> net_lwm2m_rd_client: Update Done
[06:00:38.160,430] <dbg> net_lwm2m_message_handling: lwm2m_udp_receive: reply 0x20015674 handled and removed
[06:00:42.404,296] <dbg> app_lwm2m_client: rd_client_event: Queue mode RX window closed
[06:01:00.259,613] <inf> hw_watchdog: feeding hw_watchdog
[06:01:06.260,772] <dbg> app_lwm2m_client: debug_connection_work_cb: current client_state: 3
[06:01:06.260,803] <dbg> app_lwm2m_client: debug_connection_work_cb: current retry_state: 101
[06:01:06.260,803] <dbg> app_lwm2m_client: debug_connection_work_cb: last lwm2m_rd_client_event: 10
[06:01:06.260,833] <dbg> app_lwm2m_client: debug_connection_work_cb: last lte_lc_nw_reg_status: 1
[06:01:06.260,833] <dbg> app_lwm2m_client: debug_connection_work_cb: modem_connected_to_network: 1
[06:01:06.260,864] <dbg> app_lwm2m_client: debug_connection_work_cb: current_uptime: 21666260
[06:01:06.260,894] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_reg_complete: 21637099
[06:01:06.260,925] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_reg_update: 21637402
[06:01:06.260,925] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_reg_update_complete: 21638160
[06:01:06.260,955] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_rx_off: 21642404
[06:01:06.260,955] <dbg> app_lwm2m_client: debug_connection_work_cb: ts_last_engine_suspended: 21636296
[06:01:06.261,016] <dbg> app_lwm2m_client: debug_connection_work_cb: after_last_mutex: 21637100
--- 1 messages dropped ---
The circuit board temperature value is: 26.116197

[06:01:25.242,431] <dbg> net_lwm2m_registry: lwm2m_engine_set: path:3303/0/5518, buf:0x20031840, len:8

I know that LwM2M is specifically meant for such applications, but still I don't find it very intuitive how to best handle our requirements. So here is another more general and only semi-related question that I hope you can answer for me:

We have various different sensors whose data we want to periodically send to the platform. Battery voltage, temperature, humidity, a different voltage sensor, RSS etc.

Ideally, we send a message every 12 hours (pmax) containing all that information. If the temperature value crosses a certain threshold (gt), we want to start sending every 2 hours (pmin).

Now, observing all the resources individually, they start to diverge after a day or week or month and instead of the ~7 observed values being sent at once, the device sends one value, ends transmission, drops connection and then immediately starts a new connection to send the next value. So we end up with ~7 times the energy consumption if all 7 resources are sent individually. At least that is the behaviour we have observed in the past.

I am aware of the ObserveComposite functionality, which should solve exactly that Problem, right? However, I'm not sure that our server supports the function.

So: Is ObserveComposite the only way to accomplish this behaviour, or is there a way to configure the observed resources to just send their data whenever a connection is already established?

If ObserveComposite is not an option, the three possibilities I see are:

1. Using a ReadComposite request on registration update --> higher energy consumption than necessary and chance to fail on restrictive PSM uptime. Logic on device, less flexible

2. Using the Send operation to send all the required resources --> Logic on device, less flexible

3. "Hacking" LwM2M and making a custom object with only the required resources and Observing the entire Object --> Logic on server, flexible, very ugly

I guess my question is, if there is a way to make the resources "piggy back" off of an already established connection and not triggering immediate connections a few seconds later as a consequence?

I'll of course see if I can get ObserveComposite to work, but I feel like, since LwM2M (correct me if I'm wrong) targets constrained devices with limited battery life, that I'm missing something. Shouldn't it be the default or easier to configure this "piggy back" functionality, or is it just a feature that made it into the protocol a bit later with the ObserveComposite function?

Sorry about the rant, but I hope you understand what I'm saying 🙂

RE: LwM2M client: registration SOMETIMES fails; observers not set.

SeppoTakalo — Wed, 08 May 2024 14:01:59 GMT

When LwM2M client goes into Queue mode, it stops listening, and assumes that all NAT mappings have been timed out (which happens in 30 seconds of idle period on networks that I have tested).

When it resumes, there is differences:

If CONFIG_LWM2M_TLS_SESSION_CACHING=y then it opens a new DTLS session by trying Session resumption first, and silently drops into full DTLS handshake if the resumption fails. Effect of this option that you save probably 1-2 UDP packets during the DTLS handshake.
If DTLS CID is used, then same DTLS session is kept, no new handshakes. Server learns the new IP address as it sees a familiar DTLS Connection ID.

I don't think there is issues on server side regarding this. Leshan have supported CID and Session resumption quite some time already.

I assumed originally that DTLS session caching was not enabled, but I believe it is. It does not much help in this situation.