nRF9160 Modem lock up workaround

[DamoL]

Hi, 

I've noticed an issue with the modem locking up on the nRF9160DK. I'm not sure what causes it, but it always gets err 116 - ETIMEDOUT when initialising the modem and trying to connect to a network. 

The easy fix at the moment is to unplug the DK and plug it back in. If I do, the modem connects within seconds. However, my fear is when this is in our product, it will get into this lock up state and not be able to get out of it. 

My code is loosely based on the Simple MQTT sample. When the modem_configure() function fails it has a retry delay, but without power cycling the board, it will never connect. I tried adding in lte_lc_power_off() and lte_lc_power_on() with the delay inbetween, but this didn't help. 

do {
	err = modem_configure();
	if (err) {
		printk("Retrying in %d seconds\n", CONFIG_LTE_CONNECT_RETRY_DELAY_S);
		lte_lc_power_off();
		k_sleep(K_SECONDS(CONFIG_LTE_CONNECT_RETRY_DELAY_S));
		lte_lc_power_on();
	}
} while (err);

I definitely need a soft way reset the modem when it gets in that state, and I imagine someone has also come across this issue. As for why it gets into it, I am not sure yet, but will try and understand why. 

Thanks, 

Damien

[DamoL]

Øyvind said:

Please provide the version  number on back of your DK

PCA10090 1.0.0 2021.25

I assume the modem sleep callback is telling you its in flight mode?
Look at the log when there is a successful connection. It says flight mode, then it registers.

*** Booting Zephyr OS build v2.6.0-rc1-ncs1  ***

Sensor Test Start!

The MQTT simple sample started

I: LTE Link Connecting... 

%XMODEMSLEEP: 4

%XMODEMSLEEP: 4

%XMODEMSLEEP: 4,0

+CEREG: 2,"6473","0014B715",9

+CSCON: 1

+CEREG: 5,"6473","0014B715",9,,,"11100000","00101000"

D: client_id = nrf-351358811387668

I: LTE Link Connected! 

+CEREG: 5,"6473","0014B715",9,,,"00000101","00000111"

As for trying a standard sample. Yes, and it fails to connect when in that state. A power cycle fixes it. Again, I'm pretty sure it only happens when a debug session is restarted and the modem is in PSM, but even then it doesn't happen every time.

[Øyvind]

DamoL said:

I assume the modem sleep callback is telling you its in flight mode?

Yes, this is correct, I do not expect it to be in flight mode and then connect to an LTE network. 

In order for us to figure out what is going on, we will need modem traces. This will show what is going on in the modem.

You can follow this guide or use the Trace Collect v2 (preview) found in the nRF Connect for Desktop. 

Kind regards,
Øyvind

[DamoL]

I will try and get you a trace by the end of the day. I have just spent 30+ minutes trying to get it into that state but it's connecting every time.

Is there a way I can also view the modem trace.bin file?

[Øyvind]

DamoL said:

I have just spent 30+ minutes trying to get it into that state but it's connecting every time.

That is typical 

DamoL said:

Is there a way I can also view the modem trace.bin file?

No, unfortunately, the .bin file is only readable internally. The Trace Collect v2 (preview) should provide readable Wireshark output.

[DamoL]

Hello again. 

So strangely since my last message the modem never failed to register again - until this afternoon. Now I cant seem to get it connected at all. I had made some changes in the init functions of my program, but nothing to do with the modem, so not sure why that would make a difference. 

Here is the output on my terminal - 

+CEREG: 2,"6473","0014B715",9
I: Network connection attempt timed out
I: Failed to establish LTE connection: -116
Retrying in 60 seconds
+CEREG: 0
%XMODEMSLEEP: 4
%XMODEMSLEEP: 4,0
I: LTE Link Connecting... 0
I: Failed to establish LTE connection: -120
Retrying in 60 seconds
+CEREG: 0

And I have attached the bin file from the modem trace. If you could let me know what the trace says that would be helpful.

Thanks,

Damien
trace-2021-12-08T15-24-41.944Z.bin