how to decipher HARD FAULT / MPU FAULT on zephyr running on nrf52832

[ziv123]

hello Nordic

i work on nrf52832, trying to run it with zephyr application

one board runs fine (more/less, there are some MPU FAULT issue that i am facing with no luck yet and i brought it up also in this, waiting for answer, thread: 

 performance issue with multi threadn in ncs with nrf52832 )

when i add another board i get the following fail :

ASSERTION FAIL [!arch_is_in_isr()] @ WEST_TOPDIR/zephyr/kernel/thread.c:622

  Threads may not be created in ISRs

[00000022] <err> os: ***** HARD FAULT *****
[00000022] <err> os:   Fault escalation (see below)
[00000022] [1;31m<err> os: r0/a1:  0x00000004  r1/a2:  0x0000026e  r2/a3:  0x00000000
[00000023] <err> os: r3/a4:  0x00000002 r12/ip:  0xa0000000 r14/lr:  0x0002f133
[00000023] <err> os:  xpsr:  0x6100000b
[00000023] <err> os: s[ 0]:  0x00000000  s[ 1]:  0x00000000  s[ 2]:  0x00000000  s[ 3]:  0x00000000
[00000024] <err> os: s[ 4]:  0x00000000  s[ 5]:  0x00000000  s[ 6]:  0x00000000  s[ 7]:  0x00000000
[00000024] [1;31m<err> os: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x00000000
[00000025] <err> os: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x00000000  s[15]:  0x00000000
[00000025] <err> os: fpscr:  0x00000000
[00000025] <err> os: Faulting instruction address (r15/pc): 0x000344f6
[00000025] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00000026] [1;31m<err> os: Fault during interrupt handling

[00000026] <err> os: Current thread: 0x200029d0 (unknown)
[00023642] [1;31m<err> fatal_error: Resetting system

my question is, how to decipher this log, how can i know from this lines where my problem is, which isr, which thread ?

hope to get help on this matter soon

best regards

Ziv

[Sigurd Hellesvik]

Hi Ziv,

If you set CONFIG_THREAD_NAME, the fault will usually list the thread name instead of (unknown).

I usually use this together with "CONFIG_RESET_ON_FATAL_ERROR=n" , so I do not get spammed down by fault logs.

Regards,
Sigurd Hellesvik

[ziv123]

hi Sigurd

 with the mentioned configs i see the following 

ASSERTION FAIL [z_spin_lock_valid(l)] @ WEST_TOPDIR/zephyr/include/spinlock.h:129

  Recursive spinlock 0x5a7

[00000033] <err> os: r0/a1:  0x00000004  r1/a2:  0x00000081  r2/a3:  0x00000000
[00000033] <err> os: r3/a4:  0x00000007 r12/ip:  0x80000000 r14/lr:  0x0001378d
[00000033] <err> os:  xpsr:  0x61000000
[00000034] <err> os: s[ 0]:  0x00000000  s[ 1]:  0x00000000  s[ 2]:  0x00000000  s[ 3]:  0x00000000
[00000034] <err> os: s[ 4]:  0x00000000  s[ 5]:  0x00000000  s[ 6]:  0x00000000  s[ 7]:  0x00000000
[00000035] <err> os: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x00000000
[00000035] <err> os: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x00000000  s[15]:  0x00000000
[00000035] <err> os: fpscr:  0x00000000
[00000036] [1;31m<err> os: Faulting instruction address (r15/pc): 0x000344ee
[00000036] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
[00000036] <err> os: Current thread: 0x20002af0 (main)
[00021741] <err> os: Halting system

also get this type of fail on a different branch

ASSERTION FAIL [r >= 0] @ WEST_TOPDIR/zephyr/drivers/sensor/nrf5/temp_nrf5.c:65

[00000095] <err> os: r0/a1:  0x00000004  r1/a2:  0x00000041  r2/a3:  0x20003218
[00000095] <err> os: r3/a4:  0x00000009 r12/ip:  0xfa000000 r14/lr:  0x0002ab2b
[00000095] <err> os:  xpsr:  0x41000000[0m
[00000095] <err> os: s[ 0]:  0x00000000  s[ 1]:  0x00000003  s[ 2]:  0x00000000  s[ 3]:  0x00000000
[00000095] <err> os: s[ 4]:  0x00000000  s[ 5]:  0x00000000  s[ 6]:  0x00000000  s[ 7]:  0x00000000
[00000095] <err> os: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x00000000
[00000095] <err> os: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x0000fa4d  s[15]:  0x000000fa
[00000095] [1;31m<err> os: fpscr:  0x00000000
[00000095] <err> os: Faulting instruction address (r15/pc): 0x0003a980
[00000095] [1;31m<err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
[00000095] <err> os: Current thread: 0x20001f48 (unknown)
[00000096] <err> os: Halting system

still not sure how it directs me to where the problem is, beside me thinking that this is a different type of problem then the one in the beginning of this thread

what am i missing to understand that better and how can i debug it ?

hope to read you soon

best regards

Ziv

[Sigurd Hellesvik]

Hi Ziv,

As these configurations are for debugging, it is odd that they change the way your code fail.
If you try to reproduce the issue 10 times in a row, do you always get the same issue?

In your first error in this ticket, the error says that you should not create new thread inside your interrupts. Do you create any interrupts in your own code?

Regards,
Sigurd Hellesvik

[ziv123]

hi Sigurd 

yes i do create threads in my code but not from inside an ISR 

i don't think the configs changed the type of error its just a different git branch failing on a different reason, just wanted to show that the configs did not add any input (including thread remains 'unknown'), as far as i understand, on both cases

hope to read you soon

best regards

Ziv

[Sigurd Hellesvik]

Hi Ziv,

To clarify, are you asking with help to either:

A. Solve this issue you have, or

B. Learn how you can use the Fault messages for debugging in general

?

Regards,
Sigurd Hellesvik

[ziv123]

with both A and B

A is a bit more urgent but B is crucial as well 

[Sigurd Hellesvik]

Hi,

B:
In your PROJECT/build/zephyr folder, there is a set of files.
Sometimes it is useful to have a look at the zephyr.elf file, and find the Faulting instruction address.
However, in my experience, these tools are more useful for debugging:
1. Eventual error messages or asserts from the crash.
2. Debugging functionality, see Our Debugging with VS Code tutorial.
3. Using the printk function to debug.

A:

I will help you debug some then. Lets start with some questions.

You say you get the error with one board, but not the other.
Are the two boards different?

If you try to reproduce the issue 10 times in a row, do you always get the same issue?

Do you create your own ISR?
If so, can you post the code of these (only the ISR)?

Regards,
Sigurd Hellesvik