nRF9160 reboot during FOTA image download without any crash information

Hi,

I'm implementing a FOTA update procedure for the nRF9160 using the download_client from the nRF Connect SDK 1.9.1 and modem firmware 1.3.1.

In general the image update works without any issues but in about one out of five attempts the nRF9160 reboots during the process. In this case I do not see the usual crash dump written to the console. It looks like this:

[00:01:40.844,146] <inf> download_client: Downloaded 38912/280664 bytes (13%)
[00:01:41.934,875] <inf> download_client: Downloaded 40960/280664 bytes (14%)
[00:01:42.923,980] <inf> download_client: Downloaded 43008/280664 bytes (15%)
[00:01:44.123,016] <inf> download_client: Downloaded 45056/280664 bytes (16%)
[00:01:45.847,869] <inf> download_client: Downloaded 47104/280664 bytes (16%)
*** Booting Zephyr OS build v2.7.99-ncs1-1  ***
Flash regions		Domain		Permissions
00 02 0x00000 0x18000 	Secure		rwxl
03 31 0x18000 0x100000 	Non-Secure	rwxl

Non-secure callable region 0 placed in flash region 2 with size 32.

SRAM region		Domain		Permissions
00 03 0x00000 0x08000 	Secure		rwxl
04 31 0x08000 0x40000 	Non-Secure	rwxl

I see this behaviour on nRF9160 Revision 1 and 2.

I had an Ozone debugger connected with all Vector Catches enabled in the Break&Tracepoints view during the reboot and Ozone did not catch it.

It looks like somehow the fault handler is not called. I also do not see any calls to sys_reboot() in the call path.

Any idea how to analyse this?

Thanks

Andre

Parents Reply Children
  • Hi Charlie,

    I'm testing on a single nRF9160DK and a single Actinius Icarus Bee and have seen the problem on both boards. Because of this I would assume it is a software and not a hardware problem!?

    I'm testing with a custom application we have developed, not the http_update sample you have used. I'll double check the application again.

    The application starts a Zephyr WDT but that is configured with a very long timeout and would result in a log message. Just to double check, would you see a watchdog event in the log I have sent you?

    Kind regards

    Andre

  • Hi Andre,

    asprenger said:
    The application starts a Zephyr WDT but that is configured with a very long timeout and would result in a log message. Just to double check, would you see a watchdog event in the log I have sent you?

    No, from the mode trace I can only see the modem-related activities, like the LTE connection, and IP/TCP/UDP messages. Forget to mention that in the  NCS1.9.1+MFW1.3.1+nRF9160DK HW1.1.0(nRF9160 Revision 2) tests, I add "CONFIG_LOG_DEFAULT_LEVEL=4" to enable all levels of debug information output, so you can try to do this to see if there is some debug information output before the reset. Since you add WDT in your application, it is highly suspicious that misusage of WDT causes the unexpected reset.

    Best regards,

    Charlie

  • Any news on this? I have a very similar problem with an actinius board. I configure a WDT with a very long timeout but still when I start a FOTA download the device resets itself before the download ends.

  • We stopped using the hardware WDT and use task WDTs without falling back to the hardware WDT, e.g. initialise the task WDT with: 

    task_wdt_init(NULL);

    We have a setup with several Actinius boards and have tested hourly FOTA updates with this change for the last 4 weeks without any issues.

    I would like to do a more detailed analysis of this problem but at the moment I have other priorities.

    Using the task watchdogs with the hardware WDT as backup would be the preferred solution.

    Hope that helps!

  • Thank you for the tip! I am currently testing a pretty gross solution where i use an atomic variable to keep track of the fota status, and i block the config of wdt if fota is being performed (a part from being unelegant this solution expects that the application reboots once before it can go ahead with the fota)

    the question is if task WDTs are as reliable as HW ones? I still don't have a very good understanding of the difference. In any case thank you for the answer again!

    Xavier

Related