nRF9151 modem becomes unresponsive and Link Control calls print warning lte_lc: Failed to set functional mode. Please check XSYSTEMMODE

[Atilla Byteflyer]

We are using Thingy:91 X as an evaluation platform for our upcoming product. Our test application is made up of a nested-loop structure like

while(true){
  lte_lc_connect();
  // handle possible errors

  while(true){
    //do stuff
    error = attempt_to_send();
    if(error)
      break;
   }

   lte_lc_offline();
   // wait for X minutes
}

Sproadically and seemingly randomly, the modem would get into an unoperational state. When the modem gets into this state, it cannot go out without resetting the whole device. The attempts to connect or disconnect prints the warnings

[20:46:12.849,548] <err> lte_lc: Could not get registration status, error: -1
[20:46:12.849,578]<err> lte_lc: Failed to get current registration status
[20:46:12.849,578] <err> LTE_HTTPS: Failed to connect to LTE network
[20:46:12.849,639] <err> lte_lc: Failed to set functional mode. Please check XSYSTEMMODE.
[20:46:12.849,639] <err> LTE_HTTPS: Could not disable LTE: -14

Trying to talk to the modem from the shell also fails.

at AT
Sending AT command failed with error code -1

This means we have a way of at least detecting the issue. My first question is, is there a way to reset the modem without resetting the whole device.

But more importantly, this looks like an issue in the modem hardware or firmware, so Nordic engineers should look into it.

[Achim Kraus]

Atilla Byteflyer said:

My first question is, is there a way to reset the modem without resetting the whole device.

You may try to execute a modem off/on cycle. e.g, lte_lc_power_off() and lte_lc_normal().

Atilla Byteflyer said:

this looks like an issue in the modem hardware or firmware

Not sure, it may be also an issue of the application itself.

Did you try to see, if the network rejects the modem? Either with a modem-trace or by adding a handler to the modem monitor, e.g. AT_MONITOR (or search for AT_MONITOR).

[Sebastiaan Merckx]

I am experiencing similar issues on NRF9151, modem FW 2.0.1. I have observed it at least twice, on different devices.

It's like the modem got a poweroff instruction (lte_lc_power_off) but we never send that in our application.

The application is a tracking device, that connects via LTE-M and is expected to reconnect often.

Also  lte_lc: Failed to set functional mode. Please check XSYSTEMMODE nrf9160 seems related but I couldn't really find a solution there.

After a device reset, the modem could recover in both of my cases (although it took longer for 1 case).

These are the most relevant traces I could get out of it. Including application logs. So even APN configuration fails, that really means that something bad happened to the modem right?

[33:27:50.740,142] <err> lte_lc: Could not get registration status, error: -1
[33:27:50.740,173] <inf> app_modem: LTE Link Connecting ...
[33:27:50.740,173] <inf> app_modem: Configuring APN Vodafone
[33:27:50.740,203] <err> app_modem: APN setting failed
[33:27:50.740,234] <inf> app_modem: modem low-power request ON
[33:27:50.740,264] <err> lte_lc: Failed to enable eDRX, reported error: -1
[33:27:50.740,295] <wrn> app_modem: lte_lc_edrx_req err -14
[33:27:50.740,325] <err> lte_lc: Could not get registration status, error: -1
[33:27:50.740,325] <err> lte_lc: Failed to get current registration status
[33:27:50.740,325] <wrn> app_modem: lte_lc_connect_async err -14

The device has certainly managed to make connection in the first 23 hours of uptime, but was not able in the last 10 hours, then I reset it.

[Atilla Byteflyer]

I don't have a solution yet, but a workaround. My application has a watchdog, and also periodically checks the modem status. If the modem becomes unreachable, I don't feed the watchdog and it resets the whole device. There could be a more elegant way to do it though.

/**
 * @brief Check if the modem is responsive.
 * @return true if the modem responds to the command "AT".
 */

static bool is_modem_okay(void)
{
 int err;
 char response[8]; // Expected: "OK\r\n"

 // Ping the modem to check if it is responsive
 err = nrf_modem_at_cmd(response, sizeof(response), "AT");
 if (err) {
 LOG_ERR("Modem ping failed, error: %d", err);
 return false;
 }

 return true;
}

[Sebastiaan Merckx]

Thanks for the update Atilla Byteflyer , I think I'll have to do something similar, or check for the return codes "-14" for certain instructions.