nRF9160 reboot during FOTA image download without any crash information

Hi,

I'm implementing a FOTA update procedure for the nRF9160 using the download_client from the nRF Connect SDK 1.9.1 and modem firmware 1.3.1.

In general the image update works without any issues but in about one out of five attempts the nRF9160 reboots during the process. In this case I do not see the usual crash dump written to the console. It looks like this:

[00:01:40.844,146] <inf> download_client: Downloaded 38912/280664 bytes (13%)
[00:01:41.934,875] <inf> download_client: Downloaded 40960/280664 bytes (14%)
[00:01:42.923,980] <inf> download_client: Downloaded 43008/280664 bytes (15%)
[00:01:44.123,016] <inf> download_client: Downloaded 45056/280664 bytes (16%)
[00:01:45.847,869] <inf> download_client: Downloaded 47104/280664 bytes (16%)
*** Booting Zephyr OS build v2.7.99-ncs1-1  ***
Flash regions		Domain		Permissions
00 02 0x00000 0x18000 	Secure		rwxl
03 31 0x18000 0x100000 	Non-Secure	rwxl

Non-secure callable region 0 placed in flash region 2 with size 32.

SRAM region		Domain		Permissions
00 03 0x00000 0x08000 	Secure		rwxl
04 31 0x08000 0x40000 	Non-Secure	rwxl

I see this behaviour on nRF9160 Revision 1 and 2.

I had an Ozone debugger connected with all Vector Catches enabled in the Break&Tracepoints view during the reboot and Ozone did not catch it.

It looks like somehow the fault handler is not called. I also do not see any calls to sys_reboot() in the call path.

Any idea how to analyse this?

Thanks

Andre

Parents Reply
  • We stopped using the hardware WDT and use task WDTs without falling back to the hardware WDT, e.g. initialise the task WDT with: 

    task_wdt_init(NULL);

    We have a setup with several Actinius boards and have tested hourly FOTA updates with this change for the last 4 weeks without any issues.

    I would like to do a more detailed analysis of this problem but at the moment I have other priorities.

    Using the task watchdogs with the hardware WDT as backup would be the preferred solution.

    Hope that helps!

Children
  • Thank you for the tip! I am currently testing a pretty gross solution where i use an atomic variable to keep track of the fota status, and i block the config of wdt if fota is being performed (a part from being unelegant this solution expects that the application reboots once before it can go ahead with the fota)

    the question is if task WDTs are as reliable as HW ones? I still don't have a very good understanding of the difference. In any case thank you for the answer again!

    Xavier

  • The task WDTs are based on kernel timers, so this should be a robust implementation. But they can still fail for various reasons, e.g. issues in the thread scheduler, deadlocks, resource exhaustion, etc. Therefore you want to also enable the hardware WDT as backup in case the task WDTs fail.

    The nice thing about the task WDT is you can have multiple of them, each with a different timeout, e.g. for mobile network registration, DTLS handshake, reading a sensor etc.

    So yeah, in general you want to use the together!

  • Now I am confused, does the hw wdt only have one timer? I use task_wdt_init(hw_id) to initialize it and task_wdt_add() to configure timers, are these task wdt? Then why give it the hw_id when init?

    This is how I am doing it right now and I thought I am using the hw wdt: devzone.nordicsemi.com/.../380026

Related