This post is older than 2 years and might not be relevant anymore
More Info: Consider searching for newer posts

nrf9160 modem not responding after switching to offline mode

My application uses the nrf9160 in eDRX mode. I regularly have the problem, that when switching the modem to OFFLINE mode (same for POWER-OFF mode), the modem is not responding anymore to any AT commands. I need to switch to OFFLINE mode for battery saving and for switching between Cat-M1 and Cat-NB1.

I wrote a minimalistic application for the demo board (PCA10090 0.8.5, modem Firmware 1.0.0), where I can reproduce the problem.

First the modem is configured to LTE Cat-M1 mode with an eDRX intervall of 82 seconds. Then it is set so ONLINE mode. After a connection is established or a timeout of 100 seconds expires, the modem is set to OFFLINE mode. Then this sequence is repeated.

For the application to run, the AT Command Driver and Logging with synchronous processing must be configured.

Here is the application code for the demo board:

#include <zephyr.h>
#include <stdio.h>
#include <stdlib.h>
#include <uart.h>
#include <string.h>
#include <logging/log.h>
#include <at_cmd.h>

LOG_MODULE_REGISTER(main, LOG_LEVEL_DBG);

volatile int regStatus = 0;

static void modem_NotificationHandler(char* msg)
{
  LOG_DBG("%s", msg);
  if (memcmp(msg, "+CEREG:", strlen("+CEREG:")) == 0)
  {
    char* p = msg + strlen("+CEREG:");
    regStatus = strtol(p, NULL, 10);
  }
}

void main(void)
{
  const u32_t timeout = 100;
  bool connected;
  u32_t t = 0;

	LOG_DBG("application started");

  at_cmd_set_notification_handler(modem_NotificationHandler);
  at_cmd_write("AT+CEREG=3", NULL, 0, NULL);

  while (1)
  {
    regStatus = 0;
    LOG_DBG("configuring modem");
    at_cmd_write("AT%XSYSTEMMODE=1,0,0,0", NULL, 0, NULL); // LTE Cat-M1
    at_cmd_write("AT+CEDRXS=2,4,\"0101\"", NULL, 0, NULL); // 81.92 seconds
    LOG_DBG("modem configured");
    at_cmd_write("AT+CFUN=1", NULL, 0, NULL);
    LOG_DBG("connecting . . .");
    t = 0;
    connected = false;
    while (!connected && t++ < timeout)
    {
      k_sleep(1000);
      connected = regStatus == 1 || regStatus == 5;
    }
    if (connected)
    {
      LOG_DBG("connected");
      k_sleep(10000);
    }
    else
    {
      LOG_DBG("connection timed out");
    }
    LOG_DBG("setting modem to OFFLINE");
    at_cmd_write("AT+CFUN=4", NULL, 0, NULL);
    LOG_DBG("modem set to OFFLINE");
  }
}

After the second connection cycle, when setting the modem to OFFLINE mode, the modem is not responding anymore.

Here is the output:

***** Booting Zephyr OS build v1.14.99-ncs3-snapshot2-1276-g4493a423a645 *****
[00:00:00.365,661] <dbg> main.main: application started
[00:00:00.371,734] <dbg> main.main: configuring modem
[00:00:00.379,455] <dbg> main.main: modem configured
[00:00:00.415,435] <dbg> main.main: connecting . . .
[00:00:30.800,292] <dbg> main.modem_NotificationHandler: +CEREG: 2,"0328","010D2B08",7,0,0

[00:01:05.904,235] <dbg> main.modem_NotificationHandler: +CEREG: 5,"0328","010D2B08",7

[00:01:05.912,994] <dbg> main.modem_NotificationHandler: +CEDRXP: 4,"0101","0101","0000"

[00:01:06.426,910] <dbg> main.main: connected
[00:01:16.431,854] <dbg> main.main: setting modem to OFFLINE
[00:01:16.592,407] <dbg> main.modem_NotificationHandler: +CEREG: 0,"0328","010D2B08",7,0,0

[00:01:18.002,746] <dbg> main.main: modem set to OFFLINE
[00:01:18.008,575] <dbg> main.main: configuring modem
[00:01:18.022,521] <dbg> main.main: modem configured
[00:01:18.064,727] <dbg> main.main: connecting . . .
[00:01:19.749,694] <dbg> main.modem_NotificationHandler: +CEREG: 2,"0328","01158B05",7,0,0

[00:02:58.079,345] <dbg> main.main: connection timed out
[00:02:58.085,144] <dbg> main.main: setting modem to OFFLINE

Any suggestions?

Parents
  • modem Firmware 1.0.0

    There is a MFW 1.0.1 out now for a while.  I don't know of any specific mentions in the changelog related to this, but you should probably try it out just in case.

    And I agree that this should not happen when you go OFFLINE and back, but it's actually the behavior I would expect going to POWER-OFF and back.  Nordic has confirmed that once you put a modem in POWER OFF mode you shouldn't expect it to do *anything* until after the next reboot.  Putting the modem into POWER OFF is a one-way trip.

    EDIT: We are using EDRX in our application and can go online/offline repeatedly without issue.  I'm wondering if it's somehow related to the CEDRXS or XSYSTEMMODE swtiching while offline.  We rarely do either.  The CEDRXS you can actually do while online.  One possibly related issue I've seen is that if you have other sockets open to the modem for things like TCP/UDP traffic and are not servicing them, they can block the whole subsystem and cause the AT socket to block.  But it doesn't look like you have other sockets open in this case...

  • I just checked, I am using MFW 1.0.1 on our custom board and 1.0.0 on the demo board.

    I now updated also the demo board to 1.0.1, and I still see the same behavior. Thanks for pointing that out!

    According to the AT Command Manual, the modem must be switched to OFFLINE when the XSYSTEMMODE is changed. I guess the issue is somehow connected to the eDRX mode. I also tried the CEDRX command after going ONLINE, but I does not make a difference.

    In my simplified program I do not open any other sockets as far as I see. And I see the same behavior also with the AT_CLIENT sample.

    Another Observation maybe worth noting is that for some time I still get CEREG notifications occasionally in multiples of the eDRX interval (82 seconds). So the modem is still running somehow and occasionally switches to a different cell, but is not responding…

  • Hi, any update o this issue?

    Now I'm working with modem firmware 1.2.0 and nRF SDK version 1.2, and I still have the same issue occasionally. It usually happens when the signal is rather weak, the modem probably lost connection (without notifying me about it by CEREG) and I turn the modem off with AT+CFUN=0. Then the modem does not respond. I implemented a timeout in the at_cmd_write() function, so that the driver does not hang forever. Any subsequent commands also hang / time out. Then I close and reopen the AT socket. Does not help. So a reset is the only way to get the modem working again.

    The peculiar thing is, that the modem in the background still shows 4 short moments of activity each 82 seconds, which is our eDRX interval. So it's definitely not turned off, but it's stuck.

    Out in the field, we cannot do a hard reset to bring our devices back online. Please help us with this issue.

    Best regards,

    Marcus

  • Hi Marcus,

     

    My apologies for the late reply. I have checked around in R&D, and we have not seen this behavior before. This could be a side-effect of another issue, as we have done several bug fixes in bsdlib lately: https://github.com/NordicPlayground/nrfxlib/blob/master/bsdlib/doc/CHANGELOG.rst

    Could you try to rebase your current project on master, to see if this issue still occurs?

    Could you explain a bit more about the issue at hand, is it easily reproducible, or is this something that can happen after days of running?

     

    Kind regards,

    Håkon

  • Thanks for your answer. I will try it with the master branch of the SDK.
    Unfortunately this issue is not so easy to reproduce, as it happens sporadically, but mostly after the modem appears to have lost the connection. It does not happen every time I turn the modem off or offline.
    It seems like I am not the only one having that problem: mlang3440 and lobaro-theo saw this issue too, and there are some more threads reporting on this issue.
    Best regards,
    Marcus

  • Hi Marcus,

     

    The former issue was confirmed, and seems to be slightly different in behavior compared to your issue, but that specific should have been fixed (on tag v1.1.0 and newer, should have been fixed by this PR). If the issue is not fixed on newer bsdlib versions, I think we should look into taking a modem trace, but I would first recommend that you try to reproduce this on a newer branch of nrf and bsdlib v0.7.1 to see if the problem still exist there.

     

    Kind regards,

    Håkon

Reply
  • Hi Marcus,

     

    The former issue was confirmed, and seems to be slightly different in behavior compared to your issue, but that specific should have been fixed (on tag v1.1.0 and newer, should have been fixed by this PR). If the issue is not fixed on newer bsdlib versions, I think we should look into taking a modem trace, but I would first recommend that you try to reproduce this on a newer branch of nrf and bsdlib v0.7.1 to see if the problem still exist there.

     

    Kind regards,

    Håkon

Children
  • I am seeing a similar issue and reproduces rather easily for me. I am using ncs1.3 and modem v1.2. 

    Here are the steps:


    Go online:

    lte_lc_init();
    lte_lc_system_mode_set(LTE_LC_SYSTEM_MODE_LTEM_GPS);
    lte_lc_connect_async();


    Once network is connected, connect to AWS MQTT using mqtt lib. Then send some data. Then disconnect by calling mqtt_disconnect. Once disconnected call lte_lc_offline(). This call never returns.

    Here is the at_cmd trace after calling lte_lc_offline:

    [00:00:20.072,967] <dbg> at_cmd.at_write: Sending command AT+CEREG=5
    [00:00:20.073,211] <dbg> at_cmd.at_write: Awaiting response for AT+CEREG=5
    [00:00:20.073,364] <dbg> at_cmd.socket_thread_fn: at_cmd_rx 5 bytes, OK
    
    [00:00:20.073,394] <dbg> at_cmd.at_write: Bytes sent: 10
    [00:00:20.073,425] <dbg> at_cmd.at_write: Sending command AT+CEREG?
    [00:00:20.073,486] <dbg> at_cmd.socket_thread_fn: Allocating memory slab for AT socket
    [00:00:20.073,486] <dbg> at_cmd.socket_thread_fn: Allocation done
    [00:00:20.073,638] <dbg> at_cmd.at_write: Awaiting response for AT+CEREG?
    [00:00:20.073,883] <dbg> at_cmd.socket_thread_fn: at_cmd_rx 62 bytes, +CEREG: 5,1,"D6E0","084C890D",7,,,"11100000",~
    [00:00:20.073,944] <dbg> at_cmd.at_write: Bytes sent: 9
    [00:00:20.078,430] <dbg> at_cmd.at_write: Sending command AT+CFUN=4
    [00:00:20.078,491] <dbg> at_cmd.socket_thread_fn: Allocating memory slab for AT socket
    [00:00:20.078,521] <dbg> at_cmd.socket_thread_fn: Allocation done
    [00:00:20.078,674] <dbg> at_cmd.at_write: Awaiting response for AT+CFUN=4
    [00:00:20.335,266] <dbg> at_cmd.socket_thread_fn: at_cmd_rx 58 bytes, +CEREG: 0,"D6E0","084C890D",7,0,0,"11100000",~
    [00:00:20.335,357] <dbg> at_cmd.socket_thread_fn: Allocating memory slab for AT socket
    [00:00:20.335,357] <dbg> at_cmd.socket_thread_fn: Allocation done
    [00:00:21.673,126] <dbg> at_cmd.socket_thread_fn: at_cmd_rx 12 bytes, +CSCON: 0
    
    [00:00:21.673,187] <dbg> at_cmd.socket_thread_fn: Allocating memory slab for AT socket

  • Hi ,

     

    My apologies for the late response.

    This sounds a bit more complex in terms of execution (several things happening at the same time), and will likely depend on how you handle events from mqtt as well. Could you create a new thread with detailed information related to how you reproduce this locked scenario?

     

    Kind regards,

    Håkon

Related