Modem occasionally crashes. Wont respond to AT commands

Hello,

We have devices in the field using NRF9160 V2 with NRF SDK 2.3.0 and MFW 1.3.4 on a Actinius Icarus V2 board. They are mostly working great, but sometimes some devices stop sending data. At further investigation we found out, that these devices were not able to get important information from the modem using AT commands (like IMEI, signal strength) after some random time. If the device is in this state, prompting it "AT" via UART fails and returns "<err> at_host: Error while processing AT command: -1". When resetting this device once, everything works as expected again and all AT commands work as they should.

For example one device was working for several weeks without any issues and the logs show that suddenly retrieving IMEI and signal strength fails. We are experiencing this issue on a few other devices too, it's a pattern. Problem is that it is very unpredictable and I have not been able to reproduce this issue.

The questions are:

- What happens to the device here?

- How can we get the modem out of this mode, when all communication to the modem seem to be stuck? (preferably without rebooting)

Here is a code snippet for how we get the IMEI from the modem:

void lte_module_get_imei(char* imei, size_t size){

	char response[50];
	char *next;
	struct at_param_list list;

	nrf_modem_at_cmd(response, 50, AT_CMD_IMEI_GET);
	LOG_DBG("Modem IMEI response: %s", response);

	strncpy ( imei, response, 15 );//because IMEI is 15 chars long
	imei[15]='\0';
	return;
}

A snippet of the proj.conf:

...
# LTE link control
CONFIG_LTE_LINK_CONTROL=y
CONFIG_LTE_NETWORK_MODE_LTE_M=y
CONFIG_LTE_AUTO_INIT_AND_CONNECT=n
CONFIG_LTE_NETWORK_TIMEOUT=60
## Power saving timers.
### 320 hours PSM.
CONFIG_LTE_PSM_REQ_RPTAU="11000001"
### 60 seconds active time.
CONFIG_LTE_PSM_REQ_RAT="00011110"

# Modem library
CONFIG_NRF_MODEM_LIB=y
CONFIG_NRF_MODEM_LIB_SENDMSG_BUF_SIZE=2048
CONFIG_NRF_MODEM_LIB_HEAP_SIZE=2048
CONFIG_NRF_MODEM_LIB_TRACE=y

# AT Host
CONFIG_UART_INTERRUPT_DRIVEN=y
CONFIG_AT_HOST_LIBRARY=y
...

We are using lte_lc_init_and_connect() to connect to LTE.

Also, unfortunately I cant provide a modem trace as this error is so unpredictable. If you need more information that could help, please let me know.

Maybe someone experienced a similar error pattern and knows to solve or work around it.

Any help is appreciated!

Best regards,

Moritz

Related