lwm2m_engine notifications causing modem send to lock up

RE: lwm2m_engine notifications causing modem send to lock up

HFled — Fri, 16 Apr 2021 08:22:05 GMT

The likelihood of triggering this seems to be dependant on the quality of the connection. When the connection is bad (rsrp -115db) the failure happens almost immediately.

RE: lwm2m_engine notifications causing modem send to lock up

Paul Kajdewicz — Thu, 15 Apr 2021 15:21:49 GMT

It seems, that the modem is completely unreachable after this lockup.

I periodically read out modem data via modem_info_params(). After being stuck at ''<inf> net_lwm2m_engine: Resending message: ...'', the modem_info_params() does not return anymore.

RE: lwm2m_engine notifications causing modem send to lock up

caspar.friedrich — Thu, 15 Apr 2021 14:42:37 GMT

In addition: The modem stops notifying anything after this point.

RE: lwm2m_engine notifications causing modem send to lock up

HFled — Wed, 14 Apr 2021 13:41:19 GMT

No worries!

modem version: mfw_nrf9160_1.2.3

RE: lwm2m_engine notifications causing modem send to lock up

Heidi — Wed, 14 Apr 2021 13:25:51 GMT

Hi! Apologies for the late reply.

I just need the MFW version you are using to decode the modem trace you attached, and then we can take a look.

Best regards,

Heidi

RE: lwm2m_engine notifications causing modem send to lock up

HFled — Wed, 14 Apr 2021 00:41:44 GMT

ncs commit 89427822a05a5fae1ed578fffc45145075dae8b0 "works", though it regularly produces this error:

[00:00:50.957,580] <err> net_lwm2m_engine: Poll reported a socket error, 09.
[00:00:50.957,580] <err> net_lwm2m_rd_client: RD Client socket error: 5

But at least it doesn't lock up and recovers.

ncs commit 7971211ea2d05a1a1bf5ffc5a4f3c4a728520f56 locks up, same as v1.5.0.
There's commit 0ac4f23e4f501a310fd9c2d1f46b1bb419a028d5
in betrween the two that updates the zephyr sdk from 3420cde0e37be536cda67f293784dcc1c6a92001 to 21046b8cdb4eac989c6b17bb21ffc8196be3d5e4, which I suspect is the actual culprit, but doesn't build.

Unfortunately I've not yet figured out how to bisect properly through two levels of west manifests... What I can say though is that cherry-picking all lwm2m-related changes onto 3420cde0e37be536cda67f293784dcc1c6a92001 does not cause the lock up to happen, so the issue seems to be somewhere else.

RE: lwm2m_engine notifications causing modem send to lock up

HFled — Tue, 13 Apr 2021 22:08:36 GMT

Okay, apparently it only significantly reduces the chance of this happening. When upping the number of notications to 75/5s it still happens. Still seems to be ~good with 1.4.2 (I don't see the lock up, but the fw crashes after 1-2 minutes. 1.5.0 locks up way earlier. Bisecting again.

RE: lwm2m_engine notifications causing modem send to lock up

HFled — Tue, 13 Apr 2021 12:26:07 GMT

reverting ecebb6fdb34a213af294061529ef352c4ee73927 from ncs v1.5.0 also fixes the issue in our actual fw.

RE: lwm2m_engine notifications causing modem send to lock up

HFled — Tue, 13 Apr 2021 11:35:43 GMT

I believe I have bisected the issue to this commit: github.com/.../ecebb6fdb34a213af294061529ef352c4ee73927

This is slightly surprising to me, since the code to reproduce this does not use dtls in any way, so I don't know why exactly this would change the behaviour, but it seems to. The previous commit (ae48249e1dee1510358e7871d12fe18953bbb152) works without issues.

RE: lwm2m_engine notifications causing modem send to lock up

HFled — Tue, 13 Apr 2021 09:55:56 GMT

This appears to be a regression between sdk-nrf versions 1.4.2 and 1.5.0. After changing the referenced version to 1.4.2 and adjusting the INIT_OBJ_RES_OPT call to the old signature, I don't observe the issue.