Bug in NCS v1.7.1 with 32kHz RC Oscillator and MCUboot enabled

Well, this one took me many hours of tracing to track down the root cause, so I hope it helps someone!

Note: I wasn't able to reproduce this directly with a sample included in the nRF Connect SDK. But it happens with my application, based on the "light_ctrl" mesh sample.

It occurs with the following 3-way combination (tested, only if all met);

  • CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y
  • CONFIG_BOOTLOADER_MCUBOOT=y
  • nRF Connect SDK v1.7.1 (didn't occur with v1.4.2 I last used)

I also had to enable CONFIG_BT_LL_SW_SPLIT=y  for zephyr controller to be able to fit in the flash of the nRF52833DK I'm using, but not sure if this is related (yes I intend to move to the soft device and know it is now recommended).

To fix I've changed to CONFIG_CLOCK_CONTROL_NRF_K32SRC_SYNTH=y

I've found these two issues which appear to have similar failure logs;

https://github.com/zephyrproject-rtos/zephyr/pull/25583

https://devzone.nordicsemi.com/f/nordic-q-a/76899/zephyr-build-smp_svr-for-thingy52-fails

Build error log below

Kevin

C:\freelance\git\pt118b_firmware>west build --pristine --board pt118a_nrf52833_3-1_3A --build-dir build-release_pt118a-3-1-3A -- -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=On -DNCS_TOOLCHAIN_VERSION:STRING="NONE"
-- west build: making build dir C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A pristine
-- west build: generating a build system
Including boilerplate (Zephyr base): C:/freelance/ncs/v1.7.1/zephyr/cmake/app/boilerplate.cmake
-- Application: C:/freelance/git/pt118b_firmware
-- Zephyr version: 2.6.99 (C:/freelance/ncs/v1.7.1/zephyr), build: v2.6.99-ncs1-1
-- Found Python3: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/python.exe (found suitable exact version "3.8.2") found components: Interpreter
-- Found west (found suitable version "0.11.1", minimum required is "0.7.1")
-- Board: pt118a_nrf52833_3-1_3A
-- Cache files will be written to: C:/freelance/ncs/v1.7.1/zephyr/.cache
-- Found dtc: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/dtc.exe (found suitable version "1.4.7", minimum required is "1.4.6")
-- Found toolchain: gnuarmemb (C:/freelance/ncs/v1.7.1/toolchain/opt)
-- Found BOARD.dts: C:/freelance/git/pt118b_firmware/boards/arm/pt118a_nrf52833_3-1_3A/pt118a_nrf52833_3-1_3A.dts
-- Generated zephyr.dts: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/zephyr.dts
-- Generated devicetree_unfixed.h: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/include/generated/devicetree_unfixed.h
-- Generated device_extern.h: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/include/generated/device_extern.h
-- Including generated dts.cmake file: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/dts.cmake
Parsing C:/freelance/ncs/v1.7.1/zephyr/Kconfig
Loaded configuration 'C:/freelance/git/pt118b_firmware/boards/arm/pt118a_nrf52833_3-1_3A/pt118a_nrf52833_3-1_3A_defconfig'
Merged configuration 'C:/freelance/git/pt118b_firmware/prj.conf'
Configuration saved to 'C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/.config'
Kconfig header saved to 'C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/zephyr/include/generated/autoconf.h'
-- The C compiler identification is GNU 9.2.1
-- The CXX compiler identification is GNU 9.2.1
-- The ASM compiler identification is GNU
-- Found assembler: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/arm-none-eabi-gcc.exe

=== child image mcuboot -  begin ===
Including boilerplate (Zephyr base): C:/freelance/ncs/v1.7.1/zephyr/cmake/app/boilerplate.cmake
-- Application: C:/freelance/ncs/v1.7.1/bootloader/mcuboot/boot/zephyr
-- Zephyr version: 2.6.99 (C:/freelance/ncs/v1.7.1/zephyr), build: v2.6.99-ncs1-1
-- Found Python3: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/python.exe (found suitable exact version "3.8.2") found components: Interpreter
-- Found west (found suitable version "0.11.1", minimum required is "0.7.1")
-- Board: pt118a_nrf52833_3-1_3A
-- Cache files will be written to: C:/freelance/ncs/v1.7.1/zephyr/.cache
-- Found dtc: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/dtc.exe (found suitable version "1.4.7", minimum required is "1.4.6")
-- Found toolchain: gnuarmemb (C:/freelance/ncs/v1.7.1/toolchain/opt)
-- Found BOARD.dts: C:/freelance/git/pt118b_firmware/boards/arm/pt118a_nrf52833_3-1_3A/pt118a_nrf52833_3-1_3A.dts
-- Found devicetree overlay: C:/freelance/ncs/v1.7.1/bootloader/mcuboot/boot/zephyr/dts.overlay
-- Generated zephyr.dts: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/zephyr.dts
-- Generated devicetree_unfixed.h: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/include/generated/devicetree_unfixed.h
-- Generated device_extern.h: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/include/generated/device_extern.h
-- Including generated dts.cmake file: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/dts.cmake

warning: The choice symbol BT_CTLR_TX_PWR_PLUS_8 (defined at
subsys/bluetooth\controller/Kconfig:210) was selected (set =y), but no symbol ended up as the choice
selection. See
http://docs.zephyrproject.org/latest/reference/kconfig/CONFIG_BT_CTLR_TX_PWR_PLUS_8.html and/or look
up BT_CTLR_TX_PWR_PLUS_8 in the menuconfig/guiconfig interface. The Application Development Primer,
Setting Configuration Values, and Kconfig - Tips and Best Practices sections of the manual might be
helpful too.
Parsing C:/freelance/ncs/v1.7.1/bootloader/mcuboot/boot/zephyr/Kconfig

Loaded configuration 'C:/freelance/git/pt118b_firmware/boards/arm/pt118a_nrf52833_3-1_3A/pt118a_nrf52833_3-1_3A_defconfig'
Merged configuration 'C:/freelance/ncs/v1.7.1/bootloader/mcuboot/boot/zephyr/prj.conf'
Merged configuration 'C:/freelance/ncs/v1.7.1/nrf/subsys/partition_manager/partition_manager_enabled.conf'
Configuration saved to 'C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/.config'
Kconfig header saved to 'C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/include/generated/autoconf.h'
-- The C compiler identification is GNU 9.2.1
-- The CXX compiler identification is GNU 9.2.1
-- The ASM compiler identification is GNU
-- Found assembler: C:/freelance/ncs/v1.7.1/toolchain/opt/bin/arm-none-eabi-gcc.exe
MCUBoot bootloader key file: C:/freelance/ncs/v1.7.1/bootloader/mcuboot/root-rsa-2048.pem
-- Configuring done
-- Generating done
-- Build files have been written to: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot
=== child image mcuboot -  end ===

CMake Warning at C:/freelance/ncs/v1.7.1/nrf/modules/mcuboot/CMakeLists.txt:204 (message):


        ---------------------------------------------------------
        --- WARNING: Using default MCUBoot key, it should not ---
        --- be used for production.                           ---
        ---------------------------------------------------------




-- Configuring done
-- Generating done
-- Build files have been written to: C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A
-- west build: building application
[5/319] Performing build step for 'mcuboot_subimage'
[250/257] Linking C executable zephyr\zephyr_prebuilt.elf
FAILED: zephyr/zephyr_prebuilt.elf zephyr/zephyr_prebuilt.map
cmd.exe /C "cd . && C:\freelance\ncs\v1.7.1\toolchain\opt\bin\arm-none-eabi-gcc.exe   zephyr/CMakeFiles/zephyr_prebuilt.dir/misc/empty_file.c.obj -o zephyr\zephyr_prebuilt.elf  -Wl,-T  zephyr/linker_zephyr_prebuilt.cmd  -Wl,-Map=C:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr/zephyr_prebuilt.map  -Wl,--whole-archive  app/libapp.a  zephyr/libzephyr.a  zephyr/arch/common/libarch__common.a  zephyr/arch/arch/arm/core/aarch32/libarch__arm__core__aarch32.a  zephyr/arch/arch/arm/core/aarch32/cortex_m/libarch__arm__core__aarch32__cortex_m.a  zephyr/arch/arch/arm/core/aarch32/mpu/libarch__arm__core__aarch32__mpu.a  zephyr/lib/libc/minimal/liblib__libc__minimal.a  zephyr/lib/posix/liblib__posix.a  zephyr/soc/arm/common/cortex_m/libsoc__arm__common__cortex_m.a  zephyr/soc/arm/nordic_nrf/nrf52/libsoc__arm__nordic_nrf__nrf52.a  zephyr/boards/boards/arm/pt118a_nrf52833_3-1_3A/lib..__..__..__git__pt118b_firmware__boards__arm__pt118a_nrf52833_3-1_3A.a  zephyr/drivers/gpio/libdrivers__gpio.a  zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a  zephyr/drivers/flash/libdrivers__flash.a  zephyr/drivers/serial/libdrivers__serial.a  modules/nrf/lib/fprotect/lib..__nrf__lib__fprotect.a  modules/nrf/lib/fatal_error/lib..__nrf__lib__fatal_error.a  modules/mcuboot/boot/bootutil/zephyr/libmcuboot_util.a  modules/hal_nordic/nrfx/libmodules__hal_nordic__nrfx.a  modules/mbedtls/libmodules__mbedtls.a  modules/segger/libmodules__segger.a  -Wl,--no-whole-archive  zephyr/kernel/libkernel.a  zephyr/CMakeFiles/offsets.dir/./arch/arm/core/offsets/offsets.c.obj  -L"c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/thumb/v7e-m/nofp"  -LC:/freelance/git/pt118b_firmware/build-release_pt118a-3-1-3A/mcuboot/zephyr  -lgcc  zephyr/arch/common/libisr_tables.a  -mcpu=cortex-m4  -mthumb  -mabi=aapcs  -Wl,--gc-sections  -Wl,--build-id=none  -Wl,--sort-common=descending  -Wl,--sort-section=alignment  -Wl,-u,_OffsetAbsSyms  -Wl,-u,_ConfigAbsSyms  -nostdlib  -static  -no-pie  -Wl,-X  -Wl,-N  -Wl,--orphan-handling=warn && cmd.exe /C "cd /D C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot\zephyr && C:\freelance\ncs\v1.7.1\toolchain\opt\bin\cmake.exe -E echo ""
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_sem_give':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:841: undefined reference to `z_impl_k_sem_give'
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_sem_init':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:810: undefined reference to `z_impl_k_sem_init'
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_mutex_init':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:708: undefined reference to `z_impl_k_mutex_init'
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_mutex_lock':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:724: undefined reference to `z_impl_k_mutex_lock'
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_sem_take':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:826: undefined reference to `z_impl_k_sem_take'
c:/freelance/ncs/v1.7.1/toolchain/opt/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld.exe: zephyr/drivers/sensor/nrf5/libdrivers__sensor__nrf5.a(temp_nrf5.c.obj): in function `k_mutex_unlock':
C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot/zephyr/include/generated/syscalls/kernel.h:738: undefined reference to `z_impl_k_mutex_unlock'
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
[212/319] Building C object zephyr/subsys/bluetooth/mesh/CMakeFiles/subsys__bluetooth__mesh.dir/adv.c.obj
FAILED: modules/mcuboot/mcuboot_subimage-prefix/src/mcuboot_subimage-stamp/mcuboot_subimage-build mcuboot/zephyr/zephyr.hex mcuboot/zephyr/zephyr.elf
cmd.exe /C "cd /D C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A\mcuboot && C:\freelance\ncs\v1.7.1\toolchain\opt\bin\cmake.exe --build . --"
[218/319] Building C object zephyr/subsys/bluetooth/host/CMakeFiles/subsys__bluetooth__host.dir/gatt.c.obj
ninja: build stopped: subcommand failed.
FATAL ERROR: command exited with status 1: 'C:\freelance\ncs\v1.7.1\toolchain\opt\bin\cmake.EXE' --build 'C:\freelance\git\pt118b_firmware\build-release_pt118a-3-1-3A'

  • Yes, I think (although only intuitively) that the RC clock accuracy in MCUBoot shouldn't be an issue. But could be wrong. We've not noticed anything yet after a week or two.

    I certainly think what you found could solve it - but maybe not the root cause. I think the root cause is more likely to be the calibration routine as Nordic said this is what changed, and it not being designed to support single threaded applications.

    Similar to this ticket, but for things to do with mutex/locks as in the error above instead of sleep;

    devzone.nordicsemi.com/.../zephyr-build-smp_svr-for-thingy52-fails

  • "We've not noticed anything yet after a week or two."
    Thank you for sharing this helpful info, I'll probably also solve it this way.

    "The temp_nrf5 files doesn't seem to handle calibration"
    No, it does not directly, but the calibration code needs to sample temperature from nrf's internal temperature sensor, so that's the reason.

    Yes, the root cause is that the calibration now requires a multithreading support, but the dependency between the CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC_CALIBRATION and CONFIG_MULTITHREADING config options is not defined in the code.
    It works in app builds because CONFIG_MULTITHREADING is turned on by default and it doesn't work in mcuboot builds because mcuboot's config explicitly disables it Slight smile

Related