nRF9160 Hard fault on boot

[Tron]

I'm compiliing against version 1.1.0 of the ncs, and my device have the mfw 1.1.0 firmware running. 

When starting the device, I get the following output: 

[11.27 08.48.39.322]***** Booting Zephyr OS build v2.0.99-ncs1 *****
[11.27 08.48.39.322]Flash region		Domain		Permissions
[11.27 08.48.39.322]00 0x00000 0x08000 	Secure		rwxl
[11.27 08.48.39.322]01 0x08000 0x10000 	Secure		rwxl
[11.27 08.48.39.322]02 0x10000 0x18000 	Secure		rwxl
[11.27 08.48.39.511]03 0x18000 0x20000 	Non-Secure	rwxl
[11.27 08.48.39.511]04 0x20000 0x28000 	Non-Secure	rwxl
[11.27 08.48.39.511]05 0x28000 0x30000 	Non-Secure	rwxl
[11.27 08.48.39.511]06 0x30000 0x38000 	Non-Secure	rwxl
[11.27 08.48.39.511]07 0x38000 0x40000 	Non-Secure	rwxl
[11.27 08.48.39.511]08 0x40000 0x48000 	Non-Secure	rwxl
[11.27 08.48.39.511]09 0x48000 0x50000 	Non-Secure	rwxl
[11.27 08.48.39.511]10 0x50000 0x58000 	Non-Secure	rwxl
[11.27 08.48.39.511]11 0x58000 0x60000 	Non-Secure	rwxl
[11.27 08.48.39.511]12 0x60000 0x68000 	Non-Secure	rwxl
[11.27 08.48.39.511]13 0x68000 0x70000 	Non-Secure	rwxl
[11.27 08.48.39.511]14 0x70000 0x78000 	Non-Secure	rwxl
[11.27 08.48.39.511]15 0x78000 0x80000 	Non-Secure	rwxl
[11.27 08.48.39.511]16 0x80000 0x88000 	Non-Secure	rwxl
[11.27 08.48.39.511]17 0x88000 0x90000 	Non-Secure	rwxl
[11.27 08.48.39.511]18 0x90000 0x98000 	Non-Secure	rwxl
[11.27 08.48.39.511]19 0x98000 0xa0000 	Non-Secure	rwxl
[11.27 08.48.39.511]20 0xa0000 0xa8000 	Non-Secure	rwxl
[11.27 08.48.39.511]21 0xa8000 0xb0000 	Non-Secure	rwxl
[11.27 08.48.39.511]22 0xb0000 0xb8000 	Non-Secure	rwxl
[11.27 08.48.39.511]23 0xb8000 0xc0000 	Non-Secure	rwxl
[11.27 08.48.39.511]24 0xc0000 0xc8000 	Non-Secure	rwxl
[11.27 08.48.39.511]25 0xc8000 0xd0000 	Non-Secure	rwxl
[11.27 08.48.39.511]26 0xd0000 0xd8000 	Non-Secure	rwxl
[11.27 08.48.39.511]27 0xd8000 0xe0000 	Non-Secure	rwxl
[11.27 08.48.39.511]28 0xe0000 0xe8000 	Non-Secure	rwxl
[11.27 08.48.39.511]29 0xe8000 0xf0000 	Non-Secure	rwxl
[11.27 08.48.39.511]30 0xf0000 0xf8000 	Non-Secure	rwxl
[11.27 08.48.39.511]31 0xf8000 0x100000 	Non-Secure	rwxl
[11.27 08.48.39.511]Non-secure callable region 0 placed in flash region 2 with size 32.
[11.27 08.48.39.511]
[11.27 08.48.39.620]
[11.27 08.48.39.620]SRAM region		Domain		Permissions
[11.27 08.48.39.620]00 0x00000 0x02000	Secure		rwxl
[11.27 08.48.39.620]01 0x02000 0x04000	Secure		rwxl
[11.27 08.48.39.620]02 0x04000 0x06000	Secure		rwxl
[11.27 08.48.39.620]03 0x06000 0x08000	Secure		rwxl
[11.27 08.48.39.620]04 0x08000 0x0a000	Secure		rwxl
[11.27 08.48.39.620]05 0x0a000 0x0c000	Secure		rwxl
[11.27 08.48.39.620]06 0x0c000 0x0e000	Secure		rwxl
[11.27 08.48.39.620]07 0x0e000 0x10000	Secure		rwxl
[11.27 08.48.39.620]08 0x10000 0x12000	Non-Secure	rwxl
[11.27 08.48.39.620]09 0x12000 0x14000	Non-Secure	rwxl
[11.27 08.48.39.653]10 0x14000 0x16000	Non-Secure	rwxl
[11.27 08.48.39.653]11 0x16000 0x18000	Non-Secure	rwxl
[11.27 08.48.39.653]12 0x18000 0x1a000	Non-Secure	rwxl
[11.27 08.48.39.653]13 0x1a000 0x1c000	Non-Secure	rwxl
[11.27 08.48.39.653]14 0x1c000 0x1e000	Non-Secure	rwxl
[11.27 08.48.39.653]15 0x1e000 0x20000	Non-Secure	rwxl
[11.27 08.48.39.653]16 0x20000 0x22000	Non-Secure	rwxl
[11.27 08.48.39.653]17 0x22000 0x24000	Non-Secure	rwxl
[11.27 08.48.39.653]18 0x24000 0x26000	Non-Secure	rwxl
[11.27 08.48.39.653]19 0x26000 0x28000	Non-Secure	rwxl
[11.27 08.48.39.653]20 0x28000 0x2a000	Non-Secure	rwxl
[11.27 08.48.39.686]21 0x2a000 0x2c000	Non-Secure	rwxl
[11.27 08.48.39.686]22 0x2c000 0x2e000	Non-Secure	rwxl
[11.27 08.48.39.686]23 0x2e000 0x30000	Non-Secure	rwxl
[11.27 08.48.39.686]24 0x30000 0x32000	Non-Secure	rwxl
[11.27 08.48.39.686]25 0x32000 0x34000	Non-Secure	rwxl
[11.27 08.48.39.686]26 0x34000 0x36000	Non-Secure	rwxl
[11.27 08.48.39.686]27 0x36000 0x38000	Non-Secure	rwxl
[11.27 08.48.39.686]28 0x38000 0x3a000	Non-Secure	rwxl
[11.27 08.48.39.686]29 0x3a000 0x3c000	Non-Secure	rwxl
[11.27 08.48.39.876]30 0x3c000 0x3e000	Non-Secure	rwxl
[11.27 08.48.39.876]31 0x3e000 0x40000	Non-Secure	rwxl
[11.27 08.48.39.876]
[11.27 08.48.39.876]Peripheral		Domain		Status
[11.27 08.48.39.876]00 NRF_P0               Non-Secure	OK
[11.27 08.48.39.876]01 NRF_CLOCK            Non-Secure	OK
[11.27 08.48.39.876]02 NRF_RTC1             Non-Secure	OK
[11.27 08.48.39.876]03 NRF_NVMC             Non-Secure	OK
[11.27 08.48.39.876]04 NRF_UARTE1           Non-Secure	OK
[11.27 08.48.39.876]05 NRF_UARTE2           Secure		SKIP
[11.27 08.48.39.876]06 NRF_TWIM2            Non-Secure	OK
[11.27 08.48.39.876]07 NRF_SPIM3            Non-Secure	OK
[11.27 08.48.39.876]08 NRF_TIMER0           Non-Secure	OK
[11.27 08.48.39.876]09 NRF_TIMER1           Non-Secure	OK
[11.27 08.48.39.876]10 NRF_TIMER2           Non-Secure	OK
[11.27 08.48.39.876]11 NRF_SAADC            Non-Secure	OK
[11.27 08.48.39.876]12 NRF_PWM0             Non-Secure	OK
[11.27 08.48.39.876]13 NRF_PWM1             Non-Secure	OK
[11.27 08.48.39.876]14 NRF_PWM2             Non-Secure	OK
[11.27 08.48.39.876]15 NRF_PWM3             Non-Secure	OK
[11.27 08.48.39.876]16 NRF_WDT              Secure		SKIP
[11.27 08.48.39.876]17 NRF_IPC              Non-Secure	OK
[11.27 08.48.39.876]18 NRF_VMC              Non-Secure	OK
[11.27 08.48.39.876]19 NRF_FPU              Non-Secure	OK
[11.27 08.48.39.876]20 NRF_EGU1             Non-Secure	OK
[11.27 08.48.39.876]21 NRF_EGU2             Non-Secure	OK
[11.27 08.48.39.876]22 NRF_DPPIC            Non-Secure	OK
[11.27 08.48.39.876]23 NRF_GPIOTE1          Non-Secure	OK
[11.27 08.48.39.876]24 NRF_REGULATORS       Secure		SKIP
[11.27 08.48.39.876]
[11.27 08.48.39.876]SPM: NS image at 0x18200
[11.27 08.48.39.876]SPM: NS MSP at 0x2002b8a8
[11.27 08.48.39.876]SPM: NS reset vector at 0x2120d
[11.27 08.48.39.876]SPM: prepare to jump to Non-Secure image.
[11.27 08.48.39.876][00:00:00.487,518] <err> os: ***** HARD FAULT *****
[11.27 08.48.39.876][00:00:00.487,548] <err> os:   Fault escalation (see below)
[11.27 08.48.39.876][00:00:00.487,548] <err> os: ***** BUS FAULT *****
[11.27 08.48.39.876][00:00:00.487,579] <err> os:   Precise data bus error
[11.27 08.48.39.876][00:00:00.487,579] <err> os:   BFAR Address: 0x4001850c
[11.27 08.48.39.876][00:00:00.487,609] <err> os: r0/a1:  0x00000009  r1/a2:  0x000202e5  r2/a3:  0x40018000
[11.27 08.48.39.876][00:00:00.487,609] <err> os: r3/a4:  0x00000009 r12/ip:  0x0002120c r14/lr:  0x00038f59
[11.27 08.48.39.876][00:00:00.487,640] <err> os:  xpsr:  0x40015000
[11.27 08.48.39.876][00:00:00.487,640] <err> os: s[0]:  0x00000000  s[1]:  0x00000000  s[2]:  0x00000000  s[3]:  0x00000000
[11.27 08.48.39.876]
[11.27 08.48.39.876][00:00:00.487,670] <err> os: s[4]:  0x00000000  s[5]:  0x00000000  s[6]:  0x00000000  s[7]:  0x00000000
[11.27 08.48.39.876]
[11.27 08.48.39.920][00:00:00.487,701] <err> os: s[8]:  0x00000000  s[9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x00000000
[11.27 08.48.39.920]
[11.27 08.48.39.920][00:00:00.487,731] <err> os: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x00000000  s[15]:  0x00000000
[11.27 08.48.39.920]
[11.27 08.48.39.920][00:00:00.487,731] <err> os: fpscr:  0x2002c4d8
[11.27 08.48.39.920]
[11.27 08.48.40.089][00:00:00.487,762] <err> os: Faulting instruction address (r15/pc): 0x20029760
[11.27 08.48.40.089][00:00:00.487,762] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception
[11.27 08.48.40.089][00:00:00.487,792] <err> os: Current thread: 0x200297e0 (unknown)
[11.27 08.48.40.089][00:00:00.621,612] <err> os: Halting system

This is not very informative, and I would like pointers on how to proceed. 

[Øyvind]

Hello @tronwa,

can you please provide the version of nrfjprog?  This can be found issuing nrfjprog --version

Before programming your device, can you please issue nrfjprog -e to erase your board. Then try to program your device again. 

Kind regards,
Øyvind

[Tron]

nrfjprog version: 

nrfjprog version: 10.3.0
JLinkARM.dll version: 6.54c

nrfjprog -e and reprogramming the device didn't make any changes in the output. 

[Øyvind]

Ok, then we need to move to the other addresses right after BUS FAULT:

[11.27 08.48.39.876][00:00:00.487,579] <err> os: BFAR Address: 0x4001850c
[11.27 08.48.39.876][00:00:00.487,609] <err> os: r0/a1: 0x00000009 r1/a2: 0x000202e5 r2/a3: 0x40018000
[11.27 08.48.39.876][00:00:00.487,609] <err> os: r3/a4: 0x00000009 r12/ip: 0x0002120c r14/lr: 0x00038f59

Let me know if you get anything from those.

[Tron]

I have the following addresses available: 

[11.29 11.51.56.915][00:00:00.487,854] <err> os:   Precise data bus error
[11.29 11.51.56.915][00:00:00.487,854] <err> os:   BFAR Address: 0x4001850c
[11.29 11.51.56.915][00:00:00.487,884] <err> os: r0/a1:  0x00000009  r1/a2:  0x000208d5  r2/a3:  0x40018000
[11.29 11.51.56.948][00:00:00.487,884] <err> os: r3/a4:  0x00000009 r12/ip:  0x000217fc r14/lr:  0x0003cbf5
[11.29 11.51.56.948][00:00:00.487,915] <err> os:  xpsr:  0x40015000

I have the following responses: 

C:\Users\tron>c:\gnuarmemb\bin\arm-none-eabi-addr2line.exe -e c:\nrfTest-1.1\ncs\nrf\samples\tron\udp_gps\build_01\zephyr\zephyr.elf 0x4001850c
??:0

C:\Users\tron>c:\gnuarmemb\bin\arm-none-eabi-addr2line.exe -e c:\nrfTest-1.1\ncs\nrf\samples\tron\udp_gps\build_01\zephyr\zephyr.elf 0x40018000
??:0

C:\Users\tron>c:\gnuarmemb\bin\arm-none-eabi-addr2line.exe -e c:\nrfTest-1.1\ncs\nrf\samples\tron\udp_gps\build_01\zephyr\zephyr.elf 0x00038f59
reent.c:?

C:\Users\tron>c:\gnuarmemb\bin\arm-none-eabi-addr2line.exe -e c:\nrfTest-1.1\ncs\nrf\samples\tron\udp_gps\build_01\zephyr\zephyr.elf 0x0003cbf5
C:/nrfTest-1.1/ncs/modules/hal/nordic/nrfx/drivers/src/nrfx_wdt.c:90

C:\Users\tron>c:\gnuarmemb\bin\arm-none-eabi-addr2line.exe -e c:\nrfTest-1.1\ncs\nrf\samples\tron\udp_gps\build_01\zephyr\zephyr.elf 0x40015000
??:0

I really hope this helps

[Øyvind]

Ok, this gives us something at least. Line 8 of your response, reent.c. What is this? It does not give any information about line number, so difficult to say what this is.

Line number 11 points to nrfx_wdt.c and line 90: 

nrf_wdt_reload_value_set((p_config->reload_value * 32768) / 1000);

Is the WDT enabled in your system? If so, can you please try to disable this? To see if you are able to run.

[Tron]

Yes, I need a watchdog timer; so it is enabled. Disabling it in the prj.conf file: 

# Watchdog
CONFIG_WATCHDOG=n
#CONFIG_WDT_LOG_LEVEL_DBG=n

This makes the project to run. How to proceed on enabling the WDT?

[Øyvind]

I've spoken to my colleagues. We narrowed it down to the WDT configuration not executing. and the issue is most likely that the WDT is not enabled in SPM. 

To solve this, you have to configure SPM. 

1. Navigate to your CmakeLists.txt in your project, and the following line just below the “cmake_minimum_required(…)” and save the file:
   

   set(spm_CONF_FILE
       prj.conf
       ${CMAKE_CURRENT_LIST_DIR}/spm.conf
       )
   

2. Next, you need to create the file spm.conf in your application folder
3. Add the following line to spm.conf:
   CONFIG_SPM_NRF_WDT_NS=y

Let me know how that works for you.

Kind regards,
Øyvind

[Tron]

I done what you asked, and enabled the watchdog the same way the wd was disabled, in the prj.conf file. 

It runs now, thank you for all your effort, this is not too easy debug

[Øyvind]

Glad I could help to solve the issue!