UBX devkit & Custom board not responding in bootloader or app

[Karl Andersen]

Hello NRF crew

Setup:
Bootloader: MCUBOOT

RTOS: Zephyr

NRF toolchain: v2.6.1

Dev board: ubx_bmd360eval_nrf52811

Custom board: nrf52811 mounted without a DCDC converter, no extermal crystal and a LED

Problem

For a long time my setup worked fine and i could flash both boards, with different overlays and they would work as supposed. One day my west setup in VSCode stopped working, so i started making a custom toolchain, with west, docker, make and all the neccesary dependencies, to make a Zephyr project with the NRF SDK.

This is now working and building/Compiling the project. Cool, but now i cannot get any of the boards to respond on either RTT or UART. If i enable the bootloader, the LED indicating that the bootloader is active is on and never leaves the bootloader state.

If i remove the bootloader, the app is put on the boards, but it will not respond on anything either. I have tried flashing the boards with the files (Zephyr.hex, merged.hex and app_update.bin).

To flash the boards i have tried using the GUI flash tool from NRF Connect, NRFJprog and JLink, same outcome from all of them.

I am having trouble putting a debugger on the boards, with this setup.

I am worried it is because of memory layout og some config makes it all crash. Hope someone can help me, thanks in advance.




mcuboot.conf

# Disable DCDC converter

CONFIG_BOARD_ENABLE_DCDC=n

# Use internal clock source

CONFIG_CLOCK_CONTROL_NRF_K32SRC_SYNTH=y

# Enable logging for MCUboot

CONFIG_LOG=y

CONFIG_MCUBOOT_LOG_LEVEL_INF=y

# Enable Serial Recovery over UART

CONFIG_MCUBOOT_SERIAL=y

CONFIG_BOOT_SERIAL_UART=y

# Disable UART, since Serial Recovery uses it

CONFIG_UART_CONSOLE=n

# Configure boolader to use only one slot.

CONFIG_SINGLE_APPLICATION_SLOT=y

# Turn on a LED so we can see when Serial Recovery mode is active

CONFIG_MCUBOOT_INDICATION_LED=y

# Wait before booting

CONFIG_BOOT_SERIAL_WAIT_FOR_DFU=y

CONFIG_BOOT_SERIAL_WAIT_FOR_DFU_TIMEOUT=5000

# No BOOT signature, used to remove TinyCrypt and MBEDTLS see: devzone.nordicsemi.com/.../using-zephyr-and-mcuboot-without-image-signatures

#CONFIG_BOOT_SIGNATURE_TYPE_NONE=y

prj.conf

CONFIG_GPIO=y

CONFIG_BOOTLOADER_MCUBOOT=y

# Enable logging

CONFIG_LOG=y

CONFIG_LOG_DEFAULT_LEVEL=3

# RTT console

CONFIG_USE_SEGGER_RTT=y

CONFIG_RTT_CONSOLE=y

# UART console

CONFIG_UART_CONSOLE=n

CONFIG_UART_INTERRUPT_DRIVEN=y

#Enable bluetooth subsystem

CONFIG_BT=y

CONFIG_BT_OBSERVER=y

CONFIG_BT_SCAN=y

# Enable Extended Scanning

CONFIG_BT_EXT_ADV=y

CONFIG_BT_CTLR_PHY_CODED=y

CONFIG_BT_EXT_SCAN_BUF_SIZE=1650

# Zephyr Bluetooth LE Controller needs 16 event buffers to generate Extended

# Advertising Report for receiving the complete 1650 bytes of data

CONFIG_BT_BUF_EVT_RX_COUNT=8

# Set maximum scan data length for Extended Scanning in Bluetooth LE Controller

CONFIG_BT_CTLR_SCAN_DATA_LEN_MAX=1650

# Increase Zephyr Bluetooth LE Controller Rx buffer to receive complete chain

# of PDUs

CONFIG_BT_CTLR_RX_BUFFERS=8

#Was 16

#Needed for full Extended advertising support

CONFIG_BT_LL_SW_SPLIT=y

CONFIG_RING_BUFFER=y

#Disable signed images, used to remove TinyCrypt and MBEDTLS

#CONFIG_SIGN_IMAGES=n

# Lock stack sizes

CONFIG_MAIN_STACK_SIZE=512

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=512

pm_static.yml

mcuboot: #TODO Shrink this to just the mcuboot region

address: 0x0

end_address: 0xC000 # 48 KB

placement:

before:

- mcuboot_primary

region: flash_primary

size: 0xC000

mcuboot_pad:

address: 0xC000

end_address: 0xC800

placement:

before:

- mcuboot_primary_app

region: flash_primary

size: 0x800 # 2 KB

app:

address: 0xC800

end_address: 0x2e000

placement:

align:

start: 0x1000 # Align to 4 KB boundary of flash sector

region: flash_primary

size: 0x21800 # 136 KB

mcuboot_primary:

address: 0xC000

end_address: 0x2e000

orig_span: &id001

- app

- mcuboot_pad

region: flash_primary

sharers: 0x1

size: 0x22000 # 136 KB

span: *id001

mcuboot_primary_app:

address: 0xC800

end_address: 0x2e000

orig_span: &id002

- app

region: flash_primary

size: 0x21800 # 136 KB

span: *id002

storage:

address: 0x2e000

end_address: 0x30000

region: flash_primary

size: 0x2000 # 8 KB

sram_primary:

address: 0x20000000

end_address: 0x20006000

region: sram_primary

size: 0x6000 # 24 KB

 

[Edvin]

Hello,

Did you have a bootloader before your original setup started failing?

How did it start failing in the first place?

I am having trouble putting a debugger on the boards, with this setup.

What does this mean? There is a debugger present on your board? And you are able to program it, right?

If i remove the bootloader, the app is put on the boards, but it will not respond on anything either. I have tried flashing the boards with the files (Zephyr.hex, merged.hex and app_update.bin).

How do you flash it? Do you use a command? VS Code?

Cool, but now i cannot get any of the boards to respond on either RTT or UART.

What tools are you trying to use to read the RTT or UART? Have you tried external terminals? J-Link RTT Viewer or Putty?

Best regards,

Edvin

[Karl Andersen]

Hello Edvin, 

Yes i was already using the MCUBOOT bootloader and it worked flawless before.

I have now mounted a debugger via arm-non-eabi GDB and Jlink GDB server.

I have tried flashing it with the NRF connect GUI, JlinkEXE and nrfjprog, all with the same outcome.

I have been using the Nrf connect vscode extension and JLinkRTTViewer for RTT view


From what i can see on the debugger my error is either a Stack or Heap problem

The configs that i use to change the behaviour is:

# Enable MPU
CONFIG_ARM_MPU=n

# Enable hardware stack protection
CONFIG_HW_STACK_PROTECTION=y
CONFIG_STACK_SENTINEL=y

[Karl Andersen]

Here are most of the different errors i meet in the GDB debugger.
Note: After all these errors, the board always reset and start from the beginning again. It seems to me, that when i change the configs shown before, that the errors change

Reading symbols from build/zephyr/zephyr.elf...
Remote debugging using localhost:2331
mpsc_pbuf_alloc (buffer=buffer@entry=0x20000560 <log_buffer>, wlen=wlen@entry=6, timeout=...)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/lib/os/mpsc_pbuf.c:366
warning: 366 /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/lib/os/mpsc_pbuf.c: No such file or directory
(gdb) bt
#0 mpsc_pbuf_alloc (buffer=buffer@entry=0x20000560 <log_buffer>, wlen=wlen@entry=6, timeout=...)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/lib/os/mpsc_pbuf.c:366
#1 0x00011272 in msg_alloc (buffer=buffer@entry=0x20000560 <log_buffer>, wlen=wlen@entry=6)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/subsys/logging/log_core.c:611
#2 0x00002c3a in z_log_msg_alloc (wlen=wlen@entry=6)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/subsys/logging/log_core.c:620
#3 0x00011342 in z_log_msg_simple_create (source=0x16430 <log_const_os>, level=level@entry=1,
data=data@entry=0x20004d1c <z_main_stack+412>, len=len@entry=1)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/subsys/logging/log_msg.c:76
#4 0x00011396 in z_impl_z_log_msg_simple_create_0 (source=<optimized out>, level=level@entry=1,
fmt=fmt@entry=0x17234 "***** HARD FAULT *****") at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/subsys/logging/log_msg.c:138
#5 0x00003ec8 in z_log_msg_simple_create_0 (fmt=0x17234 "***** HARD FAULT *****", level=1, source=<optimized out>)
at /home/kba/md_bt_sniffer/nrf_bt_app/app/build/zephyr/include/generated/syscalls/log_msg.h:38
#6 hard_fault (esf=0x20004a08 <z_interrupt_stacks+2056>, recoverable=recoverable@entry=0x20004d67 <z_main_stack+487>)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/fault.c:769
#7 0x0001172a in fault_handle (esf=esf@entry=0x20004a08 <z_interrupt_stacks+2056>, fault=fault@entry=3,
recoverable=recoverable@entry=0x20004d67 <z_main_stack+487>)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/fault.c:850
#8 0x00003fb2 in z_arm_fault (msp=<optimized out>, psp=<optimized out>, exc_return=<optimized out>,
callee_regs=<optimized out>) at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/fault.c:1119
#9 0x00004030 in z_arm_usage_fault () at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/fault_s.S:102
#10 <signal handler called>
#11 chunk_field (f=SIZE_AND_USED, c=11712, h=0x20000688 <logging_thread>)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/lib/os/heap.h:116
#12 chunk_size (c=11712, h=0x20000688 <logging_thread>) at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/lib/os/heap.h:144
#13 right_chunk (c=11712, h=0x20000688 <logging_thread>) at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/lib/os/heap.h:206
#14 split_chunks (h=0x20000688 <logging_thread>, lc=<optimized out>, rc=11712)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/lib/os/heap.c:121
#15 0x00000000 in ?? ()

Breakpoint 1, 0x0000a9a8 in SystemInit () at /home/kba/md_bt_sniffer/nrf_bt_app/modules/hal/nordic/nrfx/mdk/system_nrf52.c:189
189 in /home/kba/md_bt_sniffer/nrf_bt_app/modules/hal/nordic/nrfx/mdk/system_nrf52.c
(gdb)


(gdb) bt
#0 0x0000a702 in SystemInit () at /home/kba/md_bt_sniffer/nrf_bt_app/modules/hal/nordic/nrfx/mdk/system_nrf52.c:283
#1 0x00002318 in z_arm_reset () at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/reset.S:97
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

(gdb) bt
#0 0x000026b4 in z_arm_init_arch_hw_at_boot () at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/scb.c:140
#1 0x000025e4 in z_arm_reset () at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/reset.S:112

(gdb) bt
#0 z_arm_init_arch_hw_at_boot () at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/scb.c:112
#1 0x000023ea in z_impl_k_thread_abort (thread=0x0 <z_impl_k_str_out>)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/thread_abort.c:28
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

z_arm_clear_arm_mpu_config () at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/scb.c:60
warning: 60 /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/scb.c: No such file or directory

Breakpoint 1, 0x000023c6 in z_arm_is_synchronous_svc (esf=<optimized out>)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/fault.c:731
731 in /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/fault.c

0x00002334 in usage_fault (esf=0x0 <z_impl_k_str_out>)
at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/fault.c:559
warning: 559 /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/fault.c: No such file or directory

Program received signal SIGTRAP, Trace/breakpoint trap.
z_arm_reset () at /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/reset.S:73
warning: 73 /home/kba/md_bt_sniffer/nrf_bt_app/zephyr/arch/arm/core/cortex_m/reset.S: No such file or directory

[Håkon Alseth]

Hi,

 

Karl Andersen said:

#7 0x0001172a in fault_handle (esf=esf@entry=0x20004a08 <z_interrupt_stacks+2056>, fault=fault@entry=3,
recoverable=recoverable@entry=0x20004d67 <z_main_stack+487>)

in gdb:

frame 7

print /x *esf

 

This will give you the faulting regs, similar to what is outputted via the log module.

You can then use add2line on the faulting register(s) to backtrace where the fault occurred:

https://academy.nordicsemi.com/courses/nrf-connect-sdk-intermediate/lessons/lesson-2-debugging/topic/exercise-2-11/

 

mcuboot.conf

# Disable DCDC converter

CONFIG_BOARD_ENABLE_DCDC=n

Have you also disabled dcdc in prj.conf? It does not seem to set 32k configuration either in your prj.conf file.

#Needed for full Extended advertising support

CONFIG_BT_LL_SW_SPLIT=y

This defaults to the zephyr open source stack. I would strongly recommend that you use the SoftDevice Controller LL.

 

Kind regards,

Håkon