Bug in NCS v1.7.1 with 32kHz RC Oscillator and MCUboot enabled

[optimisedpd]

Well, this one took me many hours of tracing to track down the root cause, so I hope it helps someone!

Note: I wasn't able to reproduce this directly with a sample included in the nRF Connect SDK. But it happens with my application, based on the "light_ctrl" mesh sample.

It occurs with the following 3-way combination (tested, only if all met);

• CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
• CONFIG_BOOTLOADER_MCUBOOT=y
• nRF Connect SDK v1.7.1 (didn't occur with v1.4.2 I last used)

I also had to enable CONFIG_BT_LL_SW_SPLIT=y  for zephyr controller to be able to fit in the flash of the nRF52833DK I'm using, but not sure if this is related (yes I intend to move to the soft device and know it is now recommended).

To fix I've changed to CONFIG_CLOCK_CONTROL_NRF_K32SRC_SYNTH=y

I've found these two issues which appear to have similar failure logs;

https://github.com/zephyrproject-rtos/zephyr/pull/25583

https://devzone.nordicsemi.com/f/nordic-q-a/76899/zephyr-build-smp_svr-for-thingy52-fails

Build error log below

Kevin

C:\freelance\git\pt118b_firmware>west build --pristine --board pt118a_nrf52833_3-1_3A --build-dir build-release_pt118a-3-1-3A -- -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=On -DNCS_TOOLCHAIN_VERSION:STRING="NONE"
-- west build: making build dir C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A pristine
-- west build: generating a build system
Including boilerplate (Zephyr base): C:/freelance/ncs/v1.7.1/zephyr/cmake/app/boilerplate.cmake
-- Application: C:/freelance/git/pt118b_firmware
-- Zephyr version: 2.6.99 (C:/freelance/ncs/v1.7.1/zephyr), build: v2.6.99-ncs1-1
-- Found Python3: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/python.exe (found suitable exact version "3.8.2") found components: Interpreter
-- Found west (found suitable version "0.11.1", minimum required is "0.7.1")
-- Board: pt118a_nrf52833_3-1_3A
-- Cache files will be written to: C:/freelance/ncs/v1.7.1/zephyr/.cache
-- Found dtc: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/dtc.exe (found suitable version "1.4.7", minimum required is "1.4.6")
-- Found toolchain: gnuarmemb (C:/freelance/ncs/v1.7.1/toolchain/opt)
-- Found BOARD.dts: C:/freelance/git/pt118b_firmware/boards/arm/pt118a_nrf52833_3-1_3A/pt118a_nrf52833_3-1_3A.dts
-- Generated zephyr.dts: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/zephyr.dts
-- Generated devicetree_unfixed.h: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/include/generated/devicetree_unfixed.h
-- Generated device_extern.h: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/include/generated/device_extern.h
-- Including generated dts.cmake file: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/dts.cmake
Parsing C:/freelance/ncs/v1.7.1/zephyr/Kconfig
Loaded configuration 'C:/freelance/git/pt118b_firmware/boards/arm/pt118a_nrf52833_3-1_3A/pt118a_nrf52833_3-1_3A_defconfig'
Merged configuration 'C:/freelance/git/pt118b_firmware/prj.conf'
Configuration saved to 'C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/.config'
Kconfig header saved to 'C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/include/generated/autoconf.h'
-- The C compiler identification is GNU 9.2.1
-- The CXX compiler identification is GNU 9.2.1
-- The ASM compiler identification is GNU
-- Found assembler: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/arm-none-eabi-gcc.exe

=== child image mcuboot -  begin ===
Including boilerplate (Zephyr base): C:/freelance/ncs/v1.7.1/zephyr/cmake/app/boilerplate.cmake
-- Application: C:/freelance/ncs/v1.7.1/bootloader/mcuboot/boot/zephyr
-- Zephyr version: 2.6.99 (C:/freelance/ncs/v1.7.1/zephyr), build: v2.6.99-ncs1-1
-- Found Python3: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/python.exe (found suitable exact version "3.8.2") found components: Interpreter
-- Found west (found suitable version "0.11.1", minimum required is "0.7.1")
-- Board: pt118a_nrf52833_3-1_3A
-- Cache files will be written to: C:/freelance/ncs/v1.7.1/zephyr/.cache
-- Found dtc: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/dtc.exe (found suitable version "1.4.7", minimum required is "1.4.6")
-- Found toolchain: gnuarmemb (C:/freelance/ncs/v1.7.1/toolchain/opt)
-- Found BOARD.dts: C:/freelance/git/pt118b_firmware/boards/arm/pt118a_nrf52833_3-1_3A/pt118a_nrf52833_3-1_3A.dts
-- Found devicetree overlay: C:/freelance/ncs/v1.7.1/bootloader/mcuboot/boot/zephyr/dts.overlay
-- Generated zephyr.dts: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/zephyr.dts
-- Generated devicetree_unfixed.h: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/include/generated/devicetree_unfixed.h
-- Generated device_extern.h: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/include/generated/device_extern.h
-- Including generated dts.cmake file: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/dts.cmake

warning: The choice symbol BT_CTLR_TX_PWR_PLUS_8 (defined at
subsys/bluetooth\controller/Kconfig:210) was selected (set =y), but no symbol ended up as the choice
selection. See
http://docs.zephyrproject.org/latest/reference/kconfig/CONFIG_BT_CTLR_TX_PWR_PLUS_8.html and/or look
up BT_CTLR_TX_PWR_PLUS_8 in the menuconfig/guiconfig interface. The Application Development Primer,
Setting Configuration Values, and Kconfig - Tips and Best Practices sections of the manual might be
helpful too.
Parsing C:/freelance/ncs/v1.7.1/bootloader/mcuboot/boot/zephyr/Kconfig

Loaded configuration 'C:/freelance/git/pt118b_firmware/boards/arm/pt118a_nrf52833_3-1_3A/pt118a_nrf52833_3-1_3A_defconfig'
Merged configuration 'C:/freelance/ncs/v1.7.1/bootloader/mcuboot/boot/zephyr/prj.conf'
Merged configuration 'C:/freelance/ncs/v1.7.1/nrf/subsys/partition_manager/partition_manager_enabled.conf'
Configuration saved to 'C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/.config'
Kconfig header saved to 'C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/include/generated/autoconf.h'
-- The C compiler identification is GNU 9.2.1
-- The CXX compiler identification is GNU 9.2.1
-- The ASM compiler identification is GNU
-- Found assembler: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/arm-none-eabi-gcc.exe
MCUBoot bootloader key file: C:/freelance/ncs/v1.7.1/bootloader/mcuboot/root-rsa-2048.pem
-- Configuring done
-- Generating done
-- Build files have been written to: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot
=== child image mcuboot -  end ===

CMake Warning at C:/freelance/ncs/v1.7.1/nrf/modules/mcuboot/CMakeLists.txt:204 (message):


        ---------------------------------------------------------
        --- WARNING: Using default MCUBoot key, it should not ---
        --- be used for production.                           ---
        ---------------------------------------------------------




-- Configuring done
-- Generating done
-- Build files have been written to: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A
-- west build: building application
[5/319] Performing build step for 'mcuboot_subimage'
[250/257] Linking C executable zephyr\zephyr_prebuilt.elf
FAILED: zephyr/zephyr_prebuilt.elf zephyr/zephyr_prebuilt.map
cmd.exe /C "cd . && C:\freelance\ncs\v1.7.1\toolchain\opt\bin\arm-none-eabi-gcc.exe   zephyr/CMakeFiles/zephyr_prebuilt.dir/misc/empty_file.c.obj -o zephyr\zephyr_prebuilt.elf  -Wl,-T  zephyr/linker_zephyr_prebuilt.cmd  -Wl,-Map=C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/zephyr_prebuilt.map  -Wl,--whole-archive  app/libapp.a  zephyr/libzephyr.a  zephyr/arch/common/libarch__common.a  zephyr/arch/arch/arm/core/aarch32/libarch__arm__core__aarch32.a  zephyr/arch/arch/arm/core/aarch32/cortex_m/libarch__arm__core__aarch32__cortex_m.a  zephyr/arch/arch/arm/core/aarch32/mpu/libarch__arm__core__aarch32__mpu.a  zephyr/lib/libc/minimal/liblib__libc__minimal.a  zephyr/lib/posix/liblib__posix.a  zephyr/soc/arm/common/cortex_m/libsoc__arm__common__cortex_m.a  zephyr/soc/arm/nordic_nrf/nrf52/libsoc__arm__nordic_nrf__nrf52.a  zephyr/boards/boards/arm/pt118a_nrf52833_3-1_3A/lib..__..__..__git__pt118b_firmware__boards__arm__pt118a_nrf52833_3-1_3A.a  zephyr/drivers/gpio/libdrivers__gpio.a  zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a  zephyr/drivers/flash/libdrivers__flash.a  zephyr/drivers/serial/libdrivers__serial.a  modules/nrf/lib/fprotect/lib..__nrf__lib__fprotect.a  modules/nrf/lib/fatal_error/lib..__nrf__lib__fatal_error.a  modules/mcuboot/boot/bootutil/zephyr/libmcuboot_util.a  modules/hal_nordic/nrfx/libmodules__hal_nordic__nrfx.a  modules/mbedtls/libmodules__mbedtls.a  modules/segger/libmodules__segger.a  -Wl,--no-whole-archive  zephyr/kernel/libkernel.a  zephyr/CMakeFiles/offsets.dir/./arch/arm/core/offsets/offsets.c.obj  -L"c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/thumb/v7e-m/nofp"  -LC:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr  -lgcc  zephyr/arch/common/libisr_tables.a  -mcpu=cortex-m4  -mthumb  -mabi=aapcs  -Wl,--gc-sections  -Wl,--build-id=none  -Wl,--sort-common=descending  -Wl,--sort-section=alignment  -Wl,-u,_OffsetAbsSyms  -Wl,-u,_ConfigAbsSyms  -nostdlib  -static  -no-pie  -Wl,-X  -Wl,-N  -Wl,--orphan-handling=warn && cmd.exe /C "cd /D C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot\zephyr && C:\freelance\ncs\v1.7.1\toolchain\opt\bin\cmake.exe -E echo ""
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_sem_give':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:841: undefined reference to `z_impl_k_sem_give'
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_sem_init':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:810: undefined reference to `z_impl_k_sem_init'
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_mutex_init':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:708: undefined reference to `z_impl_k_mutex_init'
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_mutex_lock':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:724: undefined reference to `z_impl_k_mutex_lock'
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_sem_take':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:826: undefined reference to `z_impl_k_sem_take'
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_mutex_unlock':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:738: undefined reference to `z_impl_k_mutex_unlock'
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
[212/319] Building C object zephyr/subsys/bluetooth/mesh/CMakeFiles/subsys__bluetooth__mesh.dir/adv.c.obj
FAILED: modules/mcuboot/mcuboot_subimage-prefix/src/mcuboot_subimage-stamp/mcuboot_subimage-build mcuboot/zephyr/zephyr.hex mcuboot/zephyr/zephyr.elf
cmd.exe /C "cd /D C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot && C:\freelance\ncs\v1.7.1\toolchain\opt\bin\cmake.exe --build . --"
[218/319] Building C object zephyr/subsys/bluetooth/host/CMakeFiles/subsys__bluetooth__host.dir/gatt.c.obj
ninja: build stopped: subcommand failed.
FATAL ERROR: command exited with status 1: 'C:\freelance\ncs\v1.7.1\toolchain\opt\bin\cmake.EXE' --build 'C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A'

[Droidik]

Identical (or very similar) bug is also tracked here - https://github.com/zephyrproject-rtos/zephyr/issues/40538

[optimisedpd]

I also filed a private support case with Nordic and got some more info;

Okay, so it might not be due to a bug after all, but rather that the calibration routine for the 32kHz clock was changed (in nRFConnect SDk v1.6.0 I think). Basically, the 32kHz requires a calibration routine that depends on multithreading. So you will need to enable multithreading or disable the calibration in the sub-image that's not compiling. The temp_nrf5 files doesn't seem to handle calibration, but they do require multithreading, so please try enabling that in your application.

I would think one of the configs CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION=n or CONFIG_MULTITHREADING=y would be sufficient here

I ended up solving with CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION=n to avoid the size overhead of multithreading. However someone on the Zephyr bug does say this isn't recommended. I'm not sure why but will post there.

Note: this only needs to be set for the MCUBOOT image, not the main image, i.e. adding to a \child_image\mcuboot.conf as per https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/ug_multi_image.html which then merges with the default MCUBOOT configuration.

[Droidik]

Thank you very much for the update!
Yes, I mentioned there that it isn't recommended, because the Zephyr docs say so: "If calibration is disabled when RC is used for low frequency clock then accuracy of the low frequency clock will degrade. Disable on your own risk.".

This makes sense for a functionality that requires higher precision clock like BLE, but I guess it's OK to use it in mcuboot (that's only my impression - feel free to correct me).

I think I found the problem - https://github.com/mcu-tools/mcuboot/blob/main/boot/zephyr/Kconfig#L682

[optimisedpd]

Yes, I think (although only intuitively) that the RC clock accuracy in MCUBoot shouldn't be an issue. But could be wrong. We've not noticed anything yet after a week or two.

I certainly think what you found could solve it - but maybe not the root cause. I think the root cause is more likely to be the calibration routine as Nordic said this is what changed, and it not being designed to support single threaded applications.

Similar to this ticket, but for things to do with mutex/locks as in the error above instead of sleep;

devzone.nordicsemi.com/.../zephyr-build-smp_svr-for-thingy52-fails

[Droidik]

"We've not noticed anything yet after a week or two."
Thank you for sharing this helpful info, I'll probably also solve it this way.

"The temp_nrf5 files doesn't seem to handle calibration"
No, it does not directly, but the calibration code needs to sample temperature from nrf's internal temperature sensor, so that's the reason.

Yes, the root cause is that the calibration now requires a multithreading support, but the dependency between the CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION and CONFIG_MULTITHREADING config options is not defined in the code.
It works in app builds because CONFIG_MULTITHREADING is turned on by default and it doesn't work in mcuboot builds because mcuboot's config explicitly disables it