nRF9160 reboot during FOTA image download without any crash information

[asprenger]

Hi,

I'm implementing a FOTA update procedure for the nRF9160 using the download_client from the nRF Connect SDK 1.9.1 and modem firmware 1.3.1.

In general the image update works without any issues but in about one out of five attempts the nRF9160 reboots during the process. In this case I do not see the usual crash dump written to the console. It looks like this:

[00:01:40.844,146] <inf> download_client: Downloaded 38912/280664 bytes (13%)
[00:01:41.934,875] <inf> download_client: Downloaded 40960/280664 bytes (14%)
[00:01:42.923,980] <inf> download_client: Downloaded 43008/280664 bytes (15%)
[00:01:44.123,016] <inf> download_client: Downloaded 45056/280664 bytes (16%)
[00:01:45.847,869] <inf> download_client: Downloaded 47104/280664 bytes (16%)
*** Booting Zephyr OS build v2.7.99-ncs1-1  ***
Flash regions		Domain		Permissions
00 02 0x00000 0x18000 	Secure		rwxl
03 31 0x18000 0x100000 	Non-Secure	rwxl

Non-secure callable region 0 placed in flash region 2 with size 32.

SRAM region		Domain		Permissions
00 03 0x00000 0x08000 	Secure		rwxl
04 31 0x08000 0x40000 	Non-Secure	rwxl

I see this behaviour on nRF9160 Revision 1 and 2.

I had an Ozone debugger connected with all Vector Catches enabled in the Break&Tracepoints view during the reboot and Ozone did not catch it.

It looks like somehow the fault handler is not called. I also do not see any calls to sys_reboot() in the call path.

Any idea how to analyse this?

Thanks

Andre

[Charlie]

Hi Andre，

1) I checked the modem trace and did not find any useful information, I can only see the TCP connection was just lost due to some reason caused device to reboot. I did the following 2) 3) intensive tests to try to reproduce this issue without luck.

2) I did 10 time tests with NCS2.0.0+MFW1.3.2+nRF9160DK HW1.1.0(nRF9160 Revision 2) nRF9160: HTTP application update — nRF Connect SDK 2.0.0 documentation (nordicsemi.com) since I have it ready.

2022-06-21T14_00_21.863Z-log.txt

Then I switch to NCS1.9.1+MFW1.3.1+nRF9160DK HW1.1.0(nRF9160 Revision 2) and test nRF9160: HTTP application update — nRF Connect SDK 1.9.1 documentation (nordicsemi.com) 10 times, all passed.

2022-06-21T14_21_32.142Z-log.txt

3) Are you using a custom nRF9160 board or nRF9160DK? If it is a custom board, did you request a hardware review before? If it is nRF9160DK, how many devices you have tested by far? 

Best regards,

Charlie

[asprenger]

Hi Charlie,

I'm testing on a single nRF9160DK and a single Actinius Icarus Bee and have seen the problem on both boards. Because of this I would assume it is a software and not a hardware problem!?

I'm testing with a custom application we have developed, not the http_update sample you have used. I'll double check the application again.

The application starts a Zephyr WDT but that is configured with a very long timeout and would result in a log message. Just to double check, would you see a watchdog event in the log I have sent you?

Kind regards

Andre

[Charlie]

Hi Andre,

asprenger said:

The application starts a Zephyr WDT but that is configured with a very long timeout and would result in a log message. Just to double check, would you see a watchdog event in the log I have sent you?

No, from the mode trace I can only see the modem-related activities, like the LTE connection, and IP/TCP/UDP messages. Forget to mention that in the  NCS1.9.1+MFW1.3.1+nRF9160DK HW1.1.0(nRF9160 Revision 2) tests, I add "CONFIG_LOG_DEFAULT_LEVEL=4" to enable all levels of debug information output, so you can try to do this to see if there is some debug information output before the reset. Since you add WDT in your application, it is highly suspicious that misusage of WDT causes the unexpected reset.

Best regards,

Charlie