nRF9160 crash on assert after going offline then online

RE: nRF9160 crash on assert after going offline then online

helsing — Tue, 20 Sep 2022 10:47:02 GMT

I now have feed back from the modem team:

Memory full causes assert:
00:01:58.639038 TR_ERROR_MEMORY_FULL Failed allocation: size 2028, from heap 0080a0e8
00:01:58.639068 MDM_ERROR_ASSERT @PC:0x000d90bc

The modem supports up to three simultaneous TLS sockets. In this case the opening of a third TLS socket causes assert due to memory full. You have somehow hit an issue which consumes more memory than expected.

You would need to go over the application logic. It leaves TLS sockets open thus all reservations for memory stays there.

The application should close all unused sockets to free modem resources. The modem is indicating that PDN connection is closed (SOCKET_RPC_CLOSED_NTF_MSG) but it is the application's responsibility to close the socket.

RE: nRF9160 crash on assert after going offline then online

helsing — Mon, 19 Sep 2022 13:18:50 GMT

Thank you for the update! I will let the modem team comment and get back to you.

RE: nRF9160 crash on assert after going offline then online

Ludwig Linder — Thu, 15 Sep 2022 18:32:05 GMT

Hi, we found the issue in our code that triggers the modem crash.

We had a race condition that sometimes put the modem in flight mode before disconnecting mqtt (and therefore the socket). I figured this out because i spun up another MQTT server and on that server i instead get -12 (ENOMEM) when calling zsock_connect (inside of mqtt_connect) which according to some devzone post means that no sockets are available. (It's not in any docs I can find but it makes sense)

That means that the modem dosent clear sockets when in flight mode (which I assumed it did for some reason). It also seems that nrf_modem_lib_shutdown and nrf_modem_lib_init dosen't clear sockets either because I get continued errors after one modem crash.

But the modem crash is clearly some bug that triggers because the modem thinks it has sockets available but it actually dosent. I don't have time to figure out what causes this but it presumably is because of combination of TLS cipher suites/certs and maybe port/hostname because that is all that differed between the mqtt servers i spun up.

But you can assume our issue is fixed but I would give it to your modem team as a bug report ^^

RE: nRF9160 crash on assert after going offline then online

Mark Olsson — Thu, 15 Sep 2022 14:49:42 GMT

Our product currently uses B0 (rev1) silicon so we have to use 'mfw1.2.x'.

RE: nRF9160 crash on assert after going offline then online

helsing — Thu, 15 Sep 2022 13:55:11 GMT

We would also recommend to make the move to mfw1.3.2, unless there is any specific reason to use mfw1.2.x

RE: nRF9160 crash on assert after going offline then online

helsing — Thu, 15 Sep 2022 13:49:50 GMT

Hi Ludwig,

Would you be able to provide logs/modem trace from when using mfw1.3.2?

RE: nRF9160 crash on assert after going offline then online

Ludwig Linder — Tue, 13 Sep 2022 22:00:21 GMT

Hi Helsing, thank you. :) Tell me if theres any other info I could provide that might be of help

RE: nRF9160 crash on assert after going offline then online

helsing — Tue, 13 Sep 2022 11:33:55 GMT

Hi Ludwig,

[quote user="Ludwig Linder"]The nrf9160dk I have has a B1 nrf9160.[/quote]

Sorry, my mistake. Yes, SiP Revision B2, mfw1.3.2 and nRF Connect SDK v2.0.2 is a valid combination. I am now looking further into what could be causing the modem error.

RE: nRF9160 crash on assert after going offline then online

Ludwig Linder — Mon, 12 Sep 2022 14:02:54 GMT

My nrf9160dk is revision 2 and that's where I got the latest log from. (using mfw1.3.2)

The first post I wrote was when I was testing on a revision 1 (also called b0). However, in order to show you that this is a modem error I used my nrf9160dk b1 (also called revision 2) which I wrote in my reply to try with mfw1.3.2.

And to be clear, this nrf9160dk had version 1.2.3 before I upgraded to mfw1.3.2 and I havent downgraded it at anytime.

RE: nRF9160 crash on assert after going offline then online

helsing — Mon, 12 Sep 2022 13:35:30 GMT

Unfortunately, the modem firmware you upgraded to is not compatible with the nRF9160 SiP Revision 1. You also need to be aware that it is not recommended to downgrade to some of the older modem firmware versions. From the changelog for mfw 1.3.2:

Before applying this version:

Note that this modem firmware is targeted nRF9160 SiP Revision 2
It can be used for nRF9160 SiP Revision 1, but only for testing and development
It cannot be used for engineering samples of Revision 1 (e.g. DKs or Thingy:91 with version older than v0.9.0)
After you upgrade to MFW 1.3.x it is not recommended to downgrade to MFW 1.2.x or MFW 1.1.x.(potential file system issues)

RE: nRF9160 crash on assert after going offline then online

Ludwig Linder — Thu, 08 Sep 2022 14:59:20 GMT

The nrf9160dk I have has a B1 nrf9160.

Using mfw1.3.2 on it I instead get the following error during same circumstances:

[00:03:11.744,506] <err> modem_module: Modem error: 0x4, PC: 0xe4648

RE: nRF9160 crash on assert after going offline then online

helsing — Wed, 07 Sep 2022 14:08:23 GMT

Hi Ludwig,

Sorry I did not see this right away: It looks like you are using a combination of modem firmware and nRF Connect SDK versions that are not supported:

Please see this matrix showing compatibility: https://infocenter.nordicsemi.com/topic/comp_matrix_nrf9160/COMP/nrf9160/nrf9160_modem_fw.html?cp=2_0_3_5

In general, it is also recommended that you upgrade to revision 2(B1).

RE: nRF9160 crash on assert after going offline then online

Ludwig Linder — Mon, 05 Sep 2022 17:06:56 GMT

We are using v2.0.2 of the nRF Connect SDK.

The problem with PSM is that not all networks support it and we need to support most networks. We would be offering a very poor service if we have to pick and choose customers based on what networks are near them.

For example even the network which im testing against is not supporting PSM:

[00:00:00.802,154] <dbg> modem_module: configure_low_power: PSM requested
[00:00:00.809,906] <inf> modem_module: nRF9160 modem firmware version: mfw_nrf9160_1.2.7
[00:00:00.810,485] <inf> app_event_manager: MODEM_EVT_INITIALIZED
[00:00:00.861,511] <inf> app_event_manager: MODEM_EVT_LTE_CONNECTING
[00:00:00.861,694] <inf> modem_module: LTE attempting to connect... waiting 120 seconds before timeout
[00:00:00.861,755] <dbg> modem_module: state_set: State transition STATE_DISCONNECTED --> STATE_CONNECTING
[00:00:02.867,492] <dbg> modem_module: modem_rsrp_handler: Incoming RSRP status message, RSRP value is -96
[00:00:02.867,584] <dbg> lte_lc: at_handler_cereg: +CEREG notification: +CEREG: 2,"2724","01ADDA0B",7,0,0,"11100000","11100000"

[00:00:02.867,858] <dbg> lte_lc_helpers: parse_cereg: Network registration status: 2
[00:00:02.867,889] <dbg> lte_lc_helpers: parse_cereg: LTE mode: 7
[00:00:02.867,980] <dbg> lte_lc_helpers: event_handler_list_dispatch: Dispatching events:
[00:00:02.867,980] <dbg> lte_lc_helpers: event_handler_list_dispatch:  - handler=0x000587B9
[00:00:02.868,377] <dbg> lte_lc_helpers: event_handler_list_dispatch:  - handler=0x0003BF09
[00:00:02.868,408] <dbg> lte_lc_helpers: event_handler_list_dispatch: Done
[00:00:02.868,438] <dbg> lte_lc_helpers: event_handler_list_dispatch: Dispatching events:
[00:00:02.868,438] <dbg> lte_lc_helpers: event_handler_list_dispatch:  - handler=0x000587B9
[00:00:02.868,469] <dbg> lte_lc_helpers: event_handler_list_dispatch:  - handler=0x0003BF09
[00:00:02.868,499] <dbg> modem_module: lte_evt_handler: LTE cell changed: Cell ID: 28170763, Tracking area: 10020
[00:00:02.868,499] <dbg> lte_lc_helpers: event_handler_list_dispatch: Done
[00:00:02.868,530] <dbg> lte_lc_helpers: event_handler_list_dispatch: Dispatching events:
[00:00:02.868,530] <dbg> lte_lc_helpers: event_handler_list_dispatch:  - handler=0x000587B9
[00:00:02.868,560] <dbg> lte_lc_helpers: event_handler_list_dispatch:  - handler=0x0003BF09
[00:00:02.868,560] <dbg> lte_lc_helpers: event_handler_list_dispatch: Done
[00:00:02.868,682] <inf> app_event_manager: MODEM_EVT_LTE_RSRP_UPDATE
[00:00:02.868,835] <inf> app_event_manager: MODEM_EVT_LTE_DISCONNECTED
[00:00:02.953,704] <dbg> lte_lc: at_handler_cscon: +CSCON notification
[00:00:02.953,857] <dbg> lte_lc_helpers: event_handler_list_dispatch: Dispatching events:
[00:00:02.953,857] <dbg> lte_lc_helpers: event_handler_list_dispatch:  - handler=0x000587B9
[00:00:02.953,887] <dbg> lte_lc_helpers: event_handler_list_dispatch:  - handler=0x0003BF09
[00:00:02.953,887] <dbg> modem_module: lte_evt_handler: RRC mode: Connected
[00:00:02.953,918] <dbg> lte_lc_helpers: event_handler_list_dispatch: Done
[00:00:06.714,813] <dbg> lte_lc: at_handler_cereg: +CEREG notification: +CEREG: 5,"2724","01ADDA0B",7,,,"11100000","11100000"

Either way, putting the modem in flight mode should be supported, right? And for some reason the modem is crashing because of that.

I collected a modem trace on the nrf9160dk with firmware 1.2.3. There i get Modem error: 0x10, PC: 0xd90bc

I've noticed that the modem crash happends during TLS negotiation. Specifically when the modem is supposed to send client key exchange. I've verified traffic logs from our operator.

Attached is a modem trace from a nrf9160dk running fw1.2.3. The last thing i can see when converting to pcap is the "Certificate" sending to our server and then the ACK back. After that it crashes. Hopefully you can use the modem error and the trace to locate why?

devzone.nordicsemi.com/.../trace_2D00_2022_2D00_09_2D00_05T16_2D00_56_2D00_31.872Z.bin

RE: nRF9160 crash on assert after going offline then online

Mark Olsson — Wed, 31 Aug 2022 17:54:47 GMT

I'm working with Ludwig on this project.

We want to sleep for 24 hours unless we're woken by an external interrupt.

But the main issue is why we're getting an assert error at PC: 0xda418, could you explain what the code is trying to do at that point?

RE: nRF9160 crash on assert after going offline then online

helsing — Wed, 31 Aug 2022 13:17:37 GMT

Hi Ludwig,

What is the goal with this approach for entering sleep? Have you considered PSM (Power Saving Mode)? PSM is efficient unless you would like to sleep for a very long time. This post covers important aspects when considering PSM. (The post is three years old, so it might contain some outdated information.)

Also check out this post commenting on sleep mode in Zephyr.

Which nRF Connect SDK are you using?