Stack overflow

RE: Stack overflow

Simon — Wed, 21 Apr 2021 12:35:19 GMT

The MPU fault is due to a data access violation, however when browsing DevZone/Google/Zephyr-GitHub, a stack overflow is almost always the cause of an MPU fault in Zephyr. This is the case for you too, since "Stack overflow" was logged first, followed by the MPU Fault. I think the MPU fault happens because it tries to access an invalid memory address outside the stack.

To be honest, I don't have too much experience debugging hardfaults, but if you want to understand it better I would recommend you to take a look at the Arm Cortex-M33 documentation.

Best regards,

Simon

RE: Stack overflow

Learner — Wed, 21 Apr 2021 11:04:30 GMT

Hi Simon,

I rolled back to an old version of the firmware which ran fine without any stack crash. I then started adding gradually the new code that I had experienced the stack crash error with. I have now completed adding all the code (I think) but there is no sign of the stack crash 🙂. I am rather puzzled by this because I wanted the crash to occur again so that I can identify the line(s) of code that caused it in the first place.

I still would like you to send me more information on MPU FAULT, Stacking error (context area might be not valid) and Data Access Violation and how to debug such problems.

Thank you.

Kind regards

Mohamed

RE: Stack overflow

Simon — Mon, 19 Apr 2021 15:42:04 GMT

[quote user="Learner"]Could it be that the thread where the fault is occurring is the workqueue thread main_work_pid4?[/quote]

Yes, of course. I was way too fast to answer you earlier. My apologies for that. If the fault occured in the work item handler, it is the system work queue thread that is running and I think the CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE should get modified to increase the stack size.

If it doesn't help to increase that config, I'll get back to you tomorrow about how to to debug the MPU fault.

RE: Stack overflow

Learner — Mon, 19 Apr 2021 14:29:28 GMT

Hi Simon,

[quote userid="72692" url="~/f/nordic-q-a/72986/stack-overflow/305512#305512"]You could also set CONFIG_THREAD_NAME=y,[/quote]

Thank you. I will add the config line above in prj.conf.

I did a quick search for the string "STACK" in .config and found the following occurrences.

CONFIG_MAIN_STACK_SIZE=4096
CONFIG_PRIVILEGED_STACK_SIZE=1024
CONFIG_BT_HCI_TX_STACK_SIZE=1536
CONFIG_BT_RX_STACK_SIZE=1024
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048
CONFIG_SDC_RX_STACK_SIZE=1024
CONFIG_MPSL_SIGNAL_STACK_SIZE=1024
CONFIG_STACK_ALIGN_DOUBLE_WORD=y
CONFIG_ARM_STACK_PROTECTION=y
CONFIG_MPU_STACK_GUARD=y
CONFIG_IDLE_STACK_SIZE=320
CONFIG_ISR_STACK_SIZE=2048
CONFIG_TEST_EXTRA_STACKSIZE=0
CONFIG_HW_STACK_PROTECTION=y
CONFIG_GEN_PRIV_STACKS=y
# CONFIG_STACK_GROWS_UP is not set
CONFIG_ARCH_HAS_STACK_PROTECTION=y
CONFIG_THREAD_STACK_INFO=y
# CONFIG_INIT_STACKS is not set
# CONFIG_STACK_CANARIES is not set
CONFIG_STACK_POINTER_RANDOM=0
# CONFIG_BT_HCI_TX_STACK_SIZE_WITH_PROMPT is not set
CONFIG_BT_HCI_ECC_STACK_SIZE=1100
# CONFIG_STACK_USAGE is not set
# CONFIG_STACK_SENTINEL is not set
CONFIG_LOG_PROCESS_THREAD_STACK_SIZE=768

As you can see the main stack is indeed 4096 but there are three other stack sizes of 1024. We cam ignore CONFIG_BT_RX_STACK_SIZE=1024 because the BT code is disabled. That leaves only these two, which threads are they specifying the stack size for?

CONFIG_PRIVILEGED_STACK_SIZE=1024

CONFIG_MPSL_SIGNAL_STACK_SIZE=1024

The code that is crashing is not a sample example, it is our proprietary code under development. So, I am not allowed to share it.

Since the fault is occurring in the function k_work_handler_t pid4_tasks( void ) which is setup in main() as follows,

static struct k_work main_work_pid4;

void main( void )
{
k_work_init( &main_work_pid4, pid4_tasks );
...

while (1)
{
...

k_work_submit( &main_work_pid4 );

...

}

...

}

Could it be that the thread where the fault is occurring is the workqueue thread main_work_pid4?

Where can I find more details about MPU FAULT, Stacking error (context area might be not valid) and Data Access Violation?

I have now disabled the new code that I added recently and the fault has disappeared. So, next I am going to add the new code back in gradually and see when the fault re-appears.

Kind regards

Mohamed

RE: Stack overflow

Simon — Mon, 19 Apr 2021 11:55:34 GMT

Hi Mohamed

It may be helpful to set CONFIG_THREAD_NAME=y, then you will see the names of the threds instead of just a hex number.

It seems like the thread 0x20002860 caused the MPU fault ("Current thread: 0x20002860 (unknown)"). I think an MPU fault may be a consequence of a stack overflow, and I can see that the size of that thread is 1024 bytes ("0x20002860 : STACK: unused 348 usage 676 / 1024..."). It seems like this is the main thread (based on your findings), so I'm not sure why the size is only 1024 when you set CONFIG_MAIN_STACK_SIZE=4096. You could try to take a look at the file <sample>/build/zephyr/.config and check what CONFIG_MAIN_STACK_SIZE is equal to. You could also set CONFIG_THREAD_NAME=y, and figure out exactly which thread is causing the issue.

If this didn't help, could you upload the sample in zipped format and I'll take a look at it.

Best regards,

Simon

RE: Stack overflow

Learner — Fri, 16 Apr 2021 16:08:21 GMT

Hi Simon,

The stack overflow problem is showing its ugly head again.

The fault is occurring in the function k_work_handler_t pid4_tasks( void ) which is setup in main() as follows,

static struct k_work main_work_pid4;

void main( void )
{
k_work_init( &main_work_pid4, pid4_tasks );
...

while (1)
{
...

k_work_submit( &main_work_pid4 );

...

}

The main() stack is configured in the overlay file and so is the THREAD_ANALYZER stack,

CONFIG_MAIN_STACK_SIZE=4096

CONFIG_THREAD_ANALYZER_AUTO_STACK_SIZE=2048

The stack analyzer output is shown below in bold. Although I am configuring the main to be 4096, the stack analyzer is showing stack sizes of 2048, 1024 320 and 4096, why is this?

The stack analyzer is not showing stack usage greater than 4096. In fact, I doubled the size of the main stack but the fault is still occurring. So, it could the stack overflow is the consequence of this error "Stacking error (context area might be not valid)".

I am attaching a picture of the debugger status at the point the fault occurs.

I did not want to increase

Below is a trace capture of the Debug Terminal in SES IDE.

Please help.

Kind regards

Mohamed

*** Booting Zephyr OS build v2.4.99-ncs1-rc1 ***
Thread analyze:
0x20002780 : STACK: unused 1448 usage 600 / 2048 (29 %); CPU: 0 %
0x20002fc8 : STACK: unused 1636 usage 412 / 2048 (20 %); CPU: 0 %
0x20002db8 : STACK: unused 628 usage 396 / 1024 (38 %); CPU: 0 %
0x20002860 : STACK: unused 348 usage 676 / 1024 (66 %); CPU: 0 %
0x20002f08 : STACK: unused 288 usage 32 / 320 (10 %); CPU: 0 %
0x20002e60 : STACK: unused 3508 usage 588 / 4096 (14 %); CPU: 99 %
Thread analyze:
0x20002780 : STACK: unused 1208 usage 840 / 2048 (41 %); CPU: 0 %
0x20002fc8 : STACK: unused 1636 usage 412 / 2048 (20 %); CPU: 0 %
0x20002db8 : STACK: unused 628 usage 396 / 1024 (38 %); CPU: 0 %
0x20002860 : STACK: unused 348 usage 676 / 1024 (66 %); CPU: 0 %
0x20002f08 : STACK: unused 36 usage 284 / 320 (88 %); CPU: 19 %
0x20002e60 : STACK: unused 3508 usage 588 / 4096 (14 %); CPU: 80 %
LR1110 Driver Version: v2.0.1
LR1110 Firmware Version: HW: 22 Type: 01, FW: 03.03
System Errors = 0x20
System Errors = 0x0
Counter = 1
LR1110 Modem Packet Type = 1
Counter = 2

Thread analyze:
0x20002780 : STACK: unused 1208 usage 840 / 2048 (41 %); CPU: 0 %
0x20002fc8 : STACK: unused 1636 usage 412 / 2048 (20 %MCU TEMP = -588.24 0.000660 21.95 °C
); CPU: 0 %
--- 8 messages dropped ---
delta_time_ms = 0
Thread analyze:
0x20002780 : STACK: unused 1208 usage 840 / 2048 (41 %); CPU: 0 %
0x20002fc8 : STACK: unused 1036 usage 1012 / 2048 (49 %); CPU: 42 %
0x20002db8 : STACK: unused 628 usage 396 / 1024 (38 %); CPU: 0 %
0x20002860 : STACK: unused 300 usage 724 / 1024 (70 %); CPU: 0 %
0x20002f08 : STACK: unused 36 usage 284 / 320 (88 %); CPU: 16 %
0x20002e60 : STACK: unused 2988 usage 1108 / 4096 (27 %); CPU: 41 %
MCU TEMP = -588.24 0.000660 21.95 °C
[00:01:03.042,20[00:01:03.042,20delta_time_ms = 0

[00:01:23.802,246] <err> os: ***** MPU FAULT *****
[00:01:23.802,276] <err> os: Stacking error (context area might be not valid)
[00:01:23.802,276] <err> os: Data Access Violation
[00:01:23.802,307] <err> os: MMFAR Address: 0x200056dc
[00:01:23.802,307] <err> os: r0/a1: 0x00000000 r1/a2: 0xaaaaaaaa r2/a3: 0xe288c458
[00:01:23.802,337] <err> os: r3/a4: 0xdafdd530 r12/ip: 0x568e7d0e r14/lr: 0x3c74b9ef
[00:01:23.802,337] <err> os: xpsr: 0xf48f4a00
[00:01:23.802,368] <err> os: Faulting instruction address (r15/pc): 0x6052b7d0
[00:01:23.802,368] <err> os: >>> ZEPHYR FATAL ERROR 2: Stack overflow on CPU 0
[00:01:23.802,398] <err> os: Current thread: 0x20002860 (unknown)
[00:01:24.103,271] <err> fatal_error: Resetting system

*** Booting Zephyr OS build v2.4.99-ncs1-rc1 ***
Thread analyze:
0x20002780 : STACK: unused 1448 usage 600 / 2048 (29 %); CPU: 0 %
0x20002fc8 : STACK: unused 1636 usage 412 / 2048 (20 %); CPU: 1 %
0x20002db8 : STACK: unused 628 usage 396 / 1024 (38 %); CPU: 1 %
0x20002860 : STACK: unused 348 usage 676 / 1024 (66 %); CPU: 18 %
0x20002f08 : STACK: unused 288 usage 32 / 320 (10 %); CPU: 0 %
0x20002e60 : STACK: unused 3508 usage 588 / 4096 (14 %); CPU: 78 %
Thread analyze:
0x20002780 : STACK: unused 1208 usage 840 / 2048 (41 %); CPU: 0 %
0x20002fc8 : STACK: unused 1636 usage 412 / 2048 (20 %); CPU: 0 %
0x20002db8 : STACK: unused 628 usage 396 / 1024 (38 %); CPU: 0 %
0x20002860 : STACK: unused 348 usage 676 / 1024 (66 %); CPU: 0 %
0x20002f08 : STACK: unused 36 usage 284 / 320 (88 %); CPU: 99 %
0x20002e60 : STACK: unused 3508 usage 588 / 4096 (14 %); CPU: 0 %
LR1110 Driver Version: v2.0.1
LR1110 Firmware Version: HW: 22 Type: 01, FW: 03.03
System Errors = 0x20
System Errors = 0x0
Counter = 1

LR1110 Modem Packet Type = 1
Counter = 2

Thread analyze:
0x20002780 : STACK: unused 1208 usage 840 / 2048 (41 %); CPU: 0 %
0x20002fc8 : STACK: unused 1636 usage

RE: Stack overflow

Learner — Thu, 25 Mar 2021 09:57:25 GMT

Thank you Simon.

Yes, the Thread analyzer has proved to be very useful.

Let's just hope that it will not occur again.

Kind regards

Mohamed

RE: Stack overflow

Simon — Wed, 24 Mar 2021 21:44:40 GMT

[quote user="Learner"]Is there anything in the map file that could give me pointers when the stack size is not big enough?[/quote]

I'm not totally sure about this. You could search for the faulting address in the map file and figure out what's causing the fault stack overflow, and set a break point at that location to see what thread it's running from, and then increase the stack size of that thread.

The Thread analyzer is probably the best way of analyzing the stack usage of the threads.

[quote user="Learner"]I think changes to any configuration settings (prj.conf, overlay, Kconfig...) necessitates to re-open nRF Connect SDK project via File... I did not think SES had to be restarted.[/quote]

Yes, you are correct about this. But reopening SES follows that you run File.. again, and that might include new dts/overlay/Kconfig changes that wasn't present before you reopened SES. However, something completely different might have triggered the fault, and let's hope it doesn't show up again.

Best regards,

Simon

RE: Stack overflow

Learner — Tue, 23 Mar 2021 16:31:13 GMT

Thank you Simon.

I think changes to any configuration settings (prj.conf, overlay, Kconfig...) necessitates to re-open nRF Connect SDK project via File... I did not think SES had to be restarted.

Anyway, the problem has disappeared for now. Let's hope it will not show its ugly head again.

Is there anything in the map file that could give me pointers when the stack size is not big enough?

Kind regards

Mohamed

RE: Stack overflow

Simon — Tue, 23 Mar 2021 11:25:16 GMT

Hmm.. Maybe it was the fact that you reopened SES that changed the behaviour? Be aware that changes in the overlay or Kconfig will not be taken into effect before restarting SES (or re-running the CMake logic again in some way).

Best regards,

Simon

RE: Stack overflow

Learner — Sun, 21 Mar 2021 22:06:24 GMT

Thank you Simon for your help over the weekend. Your help is really appreciated.

It has been few days since I last rebooted my laptop. So, I rebooted it yesterday and enabled the stack analyzer module in prj.cong then started to debug my application in SES and found that in fact the stack usage is not a problem at all. I have plenty of unused stack in the main thread. The application also runs without any problems. So, it looks like my laptop was causing the problem I was seeing in SES/zephyr/application. I am rather confused as to how a problem with my laptop could cause an embedded application running on a separate target to fail with a stack overflow error. Maybe you can enlighten me and tell how this can be possible.

Thank you.

Kind regards

Mohamed

RE: Stack overflow

Simon — Sun, 21 Mar 2021 21:50:08 GMT

Could you first try to do the following?

cd <application folder>/<build_folder>/zephyr
addr2line -e zephyr.elf 0xaec010

I'm using addr2line that comes with MinGW. That command will provide you with the exact place that causes the fault. I got the address 0xaec010 from here: [00:00:41.795,074] <err> os: Faulting instruction address (r15/pc): 0x00aec010.

Then start a debug session, and put a break point on the place that caused the fault, and you will find the corresponding thread at the bottom of the call stack, like in the call stack in https://devzone.nordicsemi.com/f/nordic-q-a/70100/ecdsa-signing-crashes-when-implemented-with-bluetooth/288468#288468.

If the issue is due to a stack overflow, increasing the stack size of the thread you found should solve the problem.

There may be some more efficient ways of resolving this, but this should work.

Best regards,

Simon

RE: Stack overflow

Learner — Fri, 19 Mar 2021 23:17:08 GMT

Hi Simon,

I have had a look at the thread-analyzer link and it looks like it could help debug the stack overflow I am seeing. However, I am not sure how to use it. Can you please provide an example C code using thread_analyzer_run() and thread_analyzer_print().

Thank you.

Kind regards

Mohamed

RE: Stack overflow

Learner — Fri, 19 Mar 2021 16:41:02 GMT

Hi Simon,

I increased the stack size to 8192 in prj.conf

CONFIG_MAIN_STACK_SIZE=8192

But I am still seeing this error upon entering main and before executing any of the application instructions.

In the SES debugger, the yellow arrow points to the first curly bracket '{' in main(). I then click Go (F5) to move the cursor to the first instruction in main() then the error appears.

*** Booting Zephyr OS build v2.4.99-ncs1-rc1 ***
[00:00:41.795,043] <err> os: ***** USAGE FAULT *****
[00:00:41.795,043] <err> os: Stack overflow (context area not valid)
[00:00:41.795,043] <err> os: r0/a1: 0x245bc4ae r1/a2: 0x77fb5ffe r2/a3: 0x05821210
[00:00:41.795,074] <err> os: r3/a4: 0xbdd9cba6 r12/ip: 0x0405a049 r14/lr: 0x77faebb6
[00:00:41.795,074] <err> os: xpsr: 0x9ed6fa00
[00:00:41.795,074] <err> os: Faulting instruction address (r15/pc): 0x00aec010
[00:00:41.795,074] <err> os: >>> ZEPHYR FATAL ERROR 2: Stack overflow on CPU 0
[00:00:41.795,074] <err> os: Current thread: 0x20000668 (unknown)
[00:00:42.056,854] <err> fatal_error: Resetting system

It looks like something is going before main() starts running.

Please help.

Kind regards

Mohamed

RE: Stack overflow

Learner — Fri, 19 Mar 2021 14:59:36 GMT

Thank you Simon.

I will have a look at the suggested links.

Will the .map file reveal the source of the problem?

Kind regards

Mohamed

RE: Stack overflow

Simon — Fri, 19 Mar 2021 13:05:20 GMT

If the function is called further down the tree, but still within the main thread, you should still increase CONFIG_MAIN_STACK_SIZE. If you the function is called from within another thread, you should increas the stack size of that thread.

E.g. take a look at this DevZone case https://devzone.nordicsemi.com/f/nordic-q-a/70100/ecdsa-signing-crashes-when-implemented-with-bluetooth/288468#288468

where the hci_rx_thread was the cause and increasing CONFIG_BT_RX_STACK_SIZE solved the problem.

You may look into Ozone 3.22c as well, which makes it possible to do thread aware debugging.

These may be useful as well:

Best regards,

Simon

RE: Stack overflow

Learner — Fri, 19 Mar 2021 12:32:34 GMT

Thank you Simon.

What is the solution if the function that is causing the stack overflow is not called from main() but few levels down the function call tree?

Kind regards

Mohamed

RE: Stack overflow

Simon — Fri, 19 Mar 2021 11:47:33 GMT

If the function that caused the stack overflow is called from main, increasing CONFIG_MAIN_STACK_SIZE would be the solution then.

Best regards,

Simon