Enabling CONFIG_FPU causes stack overflow

[olegr]

Problem: stack overflow in BT threads when CONFIG_FPU is enabled.

[00:00:00.034,881] <err> os: esf_dump: r0/a1: 0x00000002 r1/a2: 0x200038dc r2/a3: 0xf0f0f0f0
[00:00:00.034,912] <err> os: esf_dump: r3/a4: 0x20004e00 r12/ip: 0x00000014 r14/lr: 0x00027aab
[00:00:00.034,942] <err> os: esf_dump: xpsr: 0x41000000
[00:00:00.034,942] <err> os: esf_dump: s[ 0]: 0x00000000 s[ 1]: 0x00000000 s[ 2]: 0x00000000 s[ 3]: 0x00000000
[00:00:00.034,973] <err> os: esf_dump: s[ 4]: 0x00000000 s[ 5]: 0x00000000 s[ 6]: 0x00000000 s[ 7]: 0x00000000
[00:00:00.034,973] <err> os: esf_dump: s[ 8]: 0x00000000 s[ 9]: 0x00000000 s[10]: 0x00000000 s[11]: 0x00000000
[00:00:00.035,003] <err> os: esf_dump: s[12]: 0x00000000 s[13]: 0x00000000 s[14]: 0x00000000 s[15]: 0x00000000
[00:00:00.035,003] <err> os: esf_dump: fpscr: 0x00000000
[00:00:00.035,034] <err> os: esf_dump: Faulting instruction address (r15/pc): 0x0001f608
[00:00:00.035,064] <err> os: z_fatal_error: >>> ZEPHYR FATAL ERROR 2: Stack overflow on CPU 0
[00:00:00.035,125] <err> os: z_fatal_error: Current thread: 0x20001c68 (BT_LW_WQ)
*** Booting Zephyr OS build v3.2.99-ncs2-rc1 ***l_error_handler: Resetting system

I'm using the basic peripheral_hr sample from the sdk and enabling CONFIG_FPU=y in prj.conf. After that board will crash several times with stack overflow in some BT thread until it manages to come up. Connecting to the peripheral then causes further crashes.

I'm using a PCA10056 devkit and vscode nrf connect extension, SDK and toolchain version 2.3.0-rc1.

My understanding that it should be possible to use FPU on nrf52840, why does this failure happen?

To reproduce the failure, create a new freestanding app using peripheral_hr and add the following to prj.conf:

CONFIG_FPU=y
CONFIG_STACK_SENTINEL=y
CONFIG_THREAD_NAME=y

[Hung Bui]

Hi Oleg, 

You may want to run thread analyzer after you enable CONFIG_FPU=y to see how much stack the application needed. 

You can find how we do that in our prj_minimal.conf in peripheral_hr: 

# In order to correctly tune the stack sizes for the threads the following
# Configurations can enabled to print the current use:
# CONFIG_THREAD_NAME=y
# CONFIG_THREAD_ANALYZER=y
# CONFIG_THREAD_ANALYZER_AUTO=y
# CONFIG_THREAD_ANALYZER_RUN_UNLOCKED=y
# CONFIG_THREAD_ANALYZER_USE_PRINTK=y
# CONFIG_THREAD_ANALYZER_AUTO_INTERVAL=20
# CONFIG_CONSOLE=y
# CONFIG_UART_CONSOLE=y
# CONFIG_SERIAL=y
# CONFIG_PRINTK=y

# Example output of thread analyzer
# BT RX               : STACK: unused 576 usage 448 / 1024 (43 %); CPU: 0 %
# BT RX pri           : STACK: unused 260 usage 188 / 448 (41 %); CPU: 0 %
# BT ECC              : STACK: unused 256 usage 888 / 1144 (77 %); CPU: 1 %
# BT TX               : STACK: unused 296 usage 344 / 640 (53 %); CPU: 0 %
# thread_analyzer     : STACK: unused 128 usage 384 / 512 (75 %); CPU: 1 %
# sysworkq            : STACK: unused 856 usage 168 / 1024 (16 %); CPU: 0 %
# logging             : STACK: unused 232 usage 536 / 768 (69 %); CPU: 0 %
# idle 00             : STACK: unused 208 usage 48 / 256 (18 %); CPU: 97 %
# main                : STACK: unused 576 usage 448 / 1024 (43 %); CPU: 0 %
CONFIG_BT_RX_STACK_SIZE=1024
CONFIG_BT_CTLR_RX_PRIO_STACK_SIZE=448
CONFIG_BT_HCI_TX_STACK_SIZE_WITH_PROMPT=y
CONFIG_BT_HCI_TX_STACK_SIZE=640
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=512
CONFIG_IDLE_STACK_SIZE=128
CONFIG_MAIN_STACK_SIZE=640
CONFIG_ISR_STACK_SIZE=1024

[olegr]

Thanks that was useful to see which threads are close to the limit. However running with thread analyzer results in sporadic crashes in the "thread_analyzer" thread with errors like

ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/lib/os/ring_buffer.c:73

[Hung Bui]

Hi Olegr, 

I'm not so sure what could be wrong here. Do you have an example that we can run here to reproduce ? 
Do you use shell or IPC UART?  There is some configuration regarding the ring buffer size here: 
https://developer.nordicsemi.com/nRF_Connect_SDK/doc/2.3.0/kconfig/index.html#!RING_BUFFER

[olegr]

Hey,

I've been using the USB UART console and have not adjusted the IPC buffer sizes from their default values.

Unfortunately I have not gotten to setting up an easy to reproduce example. However it was happening consistently with thread analyzer turned on and ceased once I disabled thread analyser.

[Hung Bui]

Hi Oleg, 
Sorry for late reply. I haven't got the time to reproduce the issue on my side. But I would assume the Thread analyzer at least now can provide you an overview of the stack requirements. Not in real time but enough information to avoid the fault ?