Sporadic Reboots During Idle Mode on nRF9160

RE: Sporadic Reboots During Idle Mode on nRF9160

Achim Kraus — Wed, 25 Sep 2024 05:38:57 GMT

I didn't recognize, that the other ticket is yours as well.

LTE/NB-IOT connection with MIKROE LTE IOT 4 CLICK

The good news:

My MIKROE LTE IOT 4 click runs stable over weeks.

I guess, your wiring and so the power supply is the issues.

For my experiments I found a USB + power supply board, which works well (see my last comment in the other ticket).

I also send you a private message. If you like, just exchange ours apps and see, how they work on the boards.

RE: Sporadic Reboots During Idle Mode on nRF9160

Luis Santos — Wed, 25 Sep 2024 02:01:06 GMT

Thank you for your reply.

As you mentioned in the previous conversation #331671, I am using too a Mikroe LTE IoT 4 Click. I reviewed the schematic and compared it with the one provided in the datasheet, and everything seems correct.

The strange part is that sometimes my application enters a restart loop, while other times the restart can take hours to happen. For instance, yesterday, the system restarted every 10 seconds and sometimes it’s not happening anymore.

When a run my application, i got the output:

*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: Current reset cause: 00000003
I: RESET_PIN
I: RESET_SOFTWARE
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
I: RRC mode: Connected
I: Network registration status: Connected - home network
I: Connected to LTE network
I: mqttInit ..
I: IPv4 Address found 20.82.16.164
I: client_id = nrf-358447177549400
I: Connection to broker using mqtt_connect
I: MQTT client connected
I: Subscribing on "stlab/down/cmd"
I: SUBACK packet id: 1234
I: RRC mode: Idle
I: Publishing "1" on "stlab/up/status"
I: RRC mode: Connected
I: RRC mode: Idle
I: Publishing "1" on "stlab/up/status"
I: RRC mode: Connected
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
I: RRC mode: Connected
I: RRC mode: Idle
I: RRC mode: Connected
I: RRC mode: Idle
I: RRC mode: Connected
I: RRC mode: Idle
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
I: RRC mode: Connected
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
I: RRC mode: Connected
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
I: No reset cause or error reading reset cause (error: 0)
I: lteInit ..
I: RRC mode: Connected
I: Network registration status: Connected - home network
I: Connected to LTE network
I: mqttInit ..
I: IPv4 Address found 20.82.16.164
I: client_id = nrf-358447177549400
I: Connection to broker using mqtt_connect
I: MQTT client connected
I: Subscribing on "stlab/down/cmd"
I: SUBACK packet id: 1234
I: RRC mode: Idle
I: Publishing "1" on "stlab/up/status"
I: RRC mode: Connected
I: RRC mode: Idle
I: Publishing "1" on "stlab/up/status"
I: RRC mode: Connected
I: RRC mode: Idle
I: Publishing "1" on "stlab/up/status"
I: RRC mode: Connected
I: RRC mode: Idle
I: Publishing "1" on "stlab/up/status"
I: RRC mode: Connected
I: RRC mode: Idle
I: Publishing "1" on "stlab/up/status"
I: RRC mode: Connected
I: RRC mode: Idle
I: Publishing "1" on "stlab/up/status"
I: RRC mode: Connected
I: RRC mode: Idle
...

The reboots don't return the code mapping.

I ran this code earlier today, but no reboot occurred like now.

I observed that when I use a 5-second interval between messages, reboots occur very sporadically. On this case, the modem doesn't go into RCC Idle.

My setup is:

Let me know if you'd like any further adjustments!

main.c

#include <stdio.h>
#include <ncs_version.h>
#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>

#include <zephyr/drivers/hwinfo.h>
#include <zephyr/drivers/watchdog.h>
#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>
#include <zephyr/logging/log_ctrl.h>
#include <zephyr/sys/__assert.h>
#include <zephyr/sys/reboot.h>

#include "lte.h"
#include "mqtt.h"

LOG_MODULE_REGISTER(main, LOG_LEVEL_INF);

static atomic_t read_reset_cause = ATOMIC_INIT(0);

static volatile uint32_t reset_cause = 0;
static volatile int32_t reset_error = 0;

uint32_t appl_reset_cause(int *flags, uint16_t *reboot_code) {
    uint32_t cause = 0;

    if (atomic_cas(&read_reset_cause, 0, 1)) {
        reset_error = hwinfo_get_reset_cause(&cause);
        if (reset_error == 0 && cause) {
            LOG_INF("Current reset cause: %08x", cause);

            if (cause & RESET_PIN) {
                LOG_INF("RESET_PIN");
            }
            if (cause & RESET_SOFTWARE) {
                LOG_INF("RESET_SOFTWARE");
            }
            if (cause & RESET_BROWNOUT) {
                LOG_INF("RESET_BROWNOUT");
            }
            if (cause & RESET_POR) {
                LOG_INF("RESET_POR");
            }
            if (cause & RESET_WATCHDOG) {
                LOG_INF("WATCHDOG");
            }
            if (cause & RESET_DEBUG) {
                LOG_INF("DEBUG");
            }
            if (cause & RESET_SECURITY) {
                LOG_INF("RESET_SECURITY");
            }
            if (cause & RESET_LOW_POWER_WAKE) {
                LOG_INF("LOWPOWER");
            }
            if (cause & RESET_CPU_LOCKUP) {
                LOG_INF("CPU");
            }
            if (cause & RESET_PARITY) {
                LOG_INF("RESET_PARITY");
            }
            if (cause & RESET_PLL) {
                LOG_INF("RESET_PLL");
            }
            if (cause & RESET_CLOCK) {
                LOG_INF("RESET_CLOCK");
            }
            if (cause & RESET_HARDWARE) {
                LOG_INF("RESET_HARDWARE");
            }
            if (cause & RESET_USER) {
                LOG_INF("RESET_USER");
            }
            if (cause & RESET_TEMPERATURE) {
                LOG_INF("RESET_TEMPERATURE");
            }
        } else {
            LOG_INF("No reset cause or error reading reset cause (error: %d)", reset_error);
        }
    }
    
    return cause;
}

int main(void) {

	int reset_cause = 0;
	uint16_t reboot_cause = 0;

	appl_reset_cause(&reset_cause, &reboot_cause);

	k_sleep(K_MSEC(100));

	lteInit();	

	k_sleep(K_MSEC(100));

	mqttInit();

	return 0;
}

RE: Sporadic Reboots During Idle Mode on nRF9160

Achim Kraus — Tue, 24 Sep 2024 05:20:34 GMT

If you have doubts, add a logging statement at line 100 and print the "cause". If that's 0, it's a POR. And ensure, that you not hwinfo_clear_reset_cause and then read it again ;-). The times in the log itself after booting are not that useful. Maybe you enable the "serial terminal" to add times, if possible (I use linux/GTKTerm, which offers that).

If that verifies, that you read 0 as reset cause, then it's a POR.

Do you use a custom board? Do you have the capacitors close to the modem? Do you supply both VDD and VDD_GPIO from the regulated 3.3V output (you may, up to a certain voltage, supply VDD direct and only VDD_GPIO via the regulated output)?

I'm not sure, why a POR happens in Idle. I would guess, it's triggered by a wakeup.

Anyway, I guess you have some ideas to investigate.

RE: Sporadic Reboots During Idle Mode on nRF9160

Luis Santos — Tue, 24 Sep 2024 03:51:39 GMT

I am now using a 5V, 1A power supply with a step-down converter to achieve a 3.3V output.

I briefly adapted your code into my main source, and here is the output I got:

#include <stdio.h>
#include <ncs_version.h>
#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>

#include <zephyr/drivers/hwinfo.h>
#include <zephyr/drivers/watchdog.h>
#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>
#include <zephyr/logging/log_ctrl.h>
#include <zephyr/sys/__assert.h>
#include <zephyr/sys/reboot.h>

#include "lte.h"
#include "mqtt.h"

LOG_MODULE_REGISTER(main, LOG_LEVEL_INF);

static atomic_t shutdown_delay = ATOMIC_INIT(-1);
static atomic_t reboot_cause = ATOMIC_INIT(-1);


static atomic_t read_reset_cause = ATOMIC_INIT(0);

static const struct device *const wdt = DEVICE_DT_GET_OR_NULL(DT_ALIAS(watchdog0));
static int wdt_channel_id = -1;

static volatile uint32_t reset_cause = 0;
static volatile int32_t reset_error = 0;

#define ERROR_CODE_INIT_NO_LTE 0x0000
#define ERROR_CODE_INIT_NO_DTLS 0x1000
#define ERROR_CODE_INIT_NO_SUCCESS 0x2000
#define ERROR_CODE_OPEN_SOCKET 0x3000
#define ERROR_CODE_TOO_MANY_FAILURES 0x4000
#define ERROR_CODE_MODEM_FAULT 0x5000
#define ERROR_CODE_REBOOT_CMD 0x6000
#define ERROR_CODE_REBOOT_MANUAL 0x7000
#define ERROR_CODE_UPDATE 0x8000
#define ERROR_CODE_LOW_VOLTAGE 0x9000
#define ERROR_CODE_REINIT_CMD 0xA000

#define ERROR_CODE(BASE, ERR) ((BASE & 0xf000) | (ERR & 0xfff))
#define ERROR_CLASS(ERR) (ERR & 0xf000)
#define ERROR_DETAIL(ERR) (ERR & 0xfff)

#define ERROR_CLASS_IS_REBOOOT(ERR) (ERROR_CLASS(ERR) == ERROR_CODE_REBOOT_CMD || ERROR_CLASS(ERR) == ERROR_CODE_REINIT_CMD)

#define FLAG_REBOOT_RETRY 1
#define FLAG_REBOOT_LOW_VOLTAGE 2
#define FLAG_REBOOT 4
#define FLAG_RESET 8
#define FLAG_POWER_ON 16

#define REBOOT_HISTORY 4
#define REBOOT_TIME_SIZE 6
#define REBOOT_CODE_SIZE sizeof(uint16_t)
#define REBOOT_SIZE (REBOOT_TIME_SIZE + REBOOT_CODE_SIZE)

static uint8_t reboot_codes[REBOOT_SIZE * REBOOT_HISTORY];

static K_MUTEX_DEFINE(settings_mutex);

int appl_settings_get_reboot_code(size_t index, int64_t *time, uint16_t *code)
{
   int res = 0;
   uint8_t buf[sizeof(reboot_codes)];

   index *= REBOOT_SIZE;
   if (index + REBOOT_SIZE <= sizeof(buf)) {
      k_mutex_lock(&settings_mutex, K_FOREVER);
      memmove(buf, reboot_codes, sizeof(reboot_codes));
      k_mutex_unlock(&settings_mutex);
      if (sys_get_be48(&buf[index])) {
         if (time) {
            uint64_t time_s = sys_get_be48(&buf[index]);
            if (time_s == 1) {
               *time = 0;
            } else {
               *time = time_s * MSEC_PER_SEC;
            }
         }
         if (code) {
            *code = sys_get_be16(&buf[index + REBOOT_TIME_SIZE]);
         }
         res = 1;
      }
   }

   return res;
}

uint32_t appl_reset_cause(int *flags, uint16_t *reboot_code)
{
   uint32_t cause = 0;
   if (atomic_cas(&read_reset_cause, 0, 1)) {
      reset_error = hwinfo_get_reset_cause(&cause);
      if (!reset_error) {
         hwinfo_clear_reset_cause();
         if (!cause) {
            // the nRF9160 uses 0 (no reset cause) to indicate POR
            uint32_t supported = 0;
            hwinfo_get_supported_reset_cause(&supported);
            if (!(supported & RESET_POR)) {
               LOG_INF("nRF9160 no reset cause, add POR");
               cause = RESET_POR;
            }
         }
         reset_cause = cause;
      }
   }
   LOG_INF("Reset cause 0x%04x", reset_cause);
   if (reset_cause) {
      // supported flags: 0x1b3
      if (reset_cause & RESET_PIN) {
         LOG_INF("PIN");
         if (flags) {
            *flags |= FLAG_RESET;
         }
      }
      if (reset_cause & RESET_SOFTWARE) {
         uint16_t code = 0;
         int rc = appl_settings_get_reboot_code(0, NULL, &code);
         int reboot = ERROR_CLASS(code);
         int detail = ERROR_DETAIL(code);
         if (rc > 0 && reboot == ERROR_CODE_TOO_MANY_FAILURES) {
            LOG_INF("Reboot 1.");
            if (!detail) {
               code = ERROR_CODE(ERROR_CODE_TOO_MANY_FAILURES, 1);
            }
            if (flags) {
               *flags |= FLAG_REBOOT_RETRY;
            }
         } else if (rc > 0 && reboot == ERROR_CODE_INIT_NO_SUCCESS) {
            LOG_INF("Reboot %u.", detail);
            if (flags) {
               *flags |= FLAG_REBOOT_RETRY;
            }
         } else if (rc > 0 && reboot == ERROR_CODE_LOW_VOLTAGE) {
            LOG_INF("Reboot low voltage.");
            if (flags) {
               *flags |= FLAG_REBOOT_LOW_VOLTAGE;
            }
         } else {
            LOG_INF("Reboot");
            if (flags) {
               *flags |= FLAG_REBOOT;
            }
         }
         if (rc > 0 && reboot_code) {
            *reboot_code = code;
         }
      }
      if (reset_cause & RESET_POR) {
         LOG_INF("Power-On");
         if (flags) {
            *flags |= FLAG_POWER_ON;
         }
      }
      if (reset_cause & RESET_WATCHDOG) {
         LOG_INF("WATCHDOG");
      }
      if (reset_cause & RESET_DEBUG) {
         LOG_INF("DEBUG");
      }
      if (reset_cause & RESET_LOW_POWER_WAKE) {
         LOG_INF("LOWPOWER");
      }
      if (reset_cause & RESET_CPU_LOCKUP) {
         LOG_INF("CPU");
      }
   } else {
      LOG_INF("none");
   }
   return reset_cause;
}


int main(void) {

	int reset_cause = 0;
	uint16_t reboot_cause = 0;

	appl_reset_cause(&reset_cause, &reboot_cause);

	k_sleep(K_MSEC(100));

	lteInit();	

	k_sleep(K_MSEC(100));

	mqttInit();

	return 0;
}

*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.253,936] <inf> main: nRF9160 no reset cause, add POR
[00:00:00.253,936] <inf> main: Reset cause 0x0008
[00:00:00.253,967] <inf> main: Power-On
[00:00:00.354,034] <inf> lte: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.253,784] <inf> main: nRF9160 no reset cause, add POR
[00:00:00.253,784] <inf> main: Reset cause 0x0008
[00:00:00.253,814] <inf> main: Power-On
[00:00:00.353,881] <inf> lte: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.253,631] <inf> main: nRF9160 no reset cause, add POR
[00:00:00.253,631] <inf> main: Reset cause 0x0008
[00:00:00.253,662] <inf> main: Power-On
[00:00:00.353,729] <inf> lte: lteInit ..
*** Booting nRF Connect SDK v2.7.0-5cb85570ca43 ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.253,814] <inf> main: nRF9160 no reset cause, add POR
[00:00:00.253,814] <inf> main: Reset cause 0x0008
[00:00:00.253,845] <inf> main: Power-On
[00:00:00.353,912] <inf> lte: lteInit ..

Could this still be happening because of a power supply issue?

◆ RESET_POR

#define RESET_POR BIT(3)

#include <zephyr/drivers/hwinfo.h>

Power-on reset (POR)

RE: Sporadic Reboots During Idle Mode on nRF9160

Achim Kraus — Sun, 22 Sep 2024 13:07:12 GMT

Such reboot are in my experience either caused by an program fault, a watchdog reset or a "power on" reset, when the power supply/battery doesn't work proper.

> 3. How can I debug this issue and pinpoint the cause of the restarts? Are there specific steps or diagnostics I should be focusing on?

The reset cause may help to narrow the scope.

prj.conf:

CONFIG_HWINFO=y

src:
hwinfo_get_reset_cause

> 1. What could be causing these reboots, particularly when the modem is in Idle mode?

Once you know, the reset cause, the answer may be easier. e.g. if it's the watchdog and you use an 60s watchdog interval, then you may fix it by extending the a little, say 70s. If it's a power on reset, then charge the battery.

> 2. Is it normal for the application to restart if there's an issue with the modem?

No, therefore I guess, there is an issue with your application.