SecureFault on NRF9160 after upgrading to HW model v2 and NCS 2.9.0

[Martin Vitek]

I have upgraded nRF Connect SDK from v2.6.0 to v2.9.0 and also upgraded custom board definition to the HW model v2 according to the nrf9160dk board files. After compiling and uploading code to the board, I'm getting error: FATAL ERROR: SecureFault. I have also tried downgrading SDK to v2.7.0 and v2.8.0, but the result was the same.

Am I missing something?

I have also spotted, that nrf9160dk_nrf9160_ns.dts defines zephyr,sram = &sram0_ns;, but nrf9161dk_nrf9161_ns.dts defines zephyr,sram = &sram0_ns_app;. Which one is correct, or does it matter?

Minimal project:

cm01test.zip

CMake log:

/usr/bin/cmake --preset cm-01 -S /home/martinv/Projekty/nextdrop/unified-fw_cm01test -B /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01
Loading Zephyr default modules (Zephyr base).Preset environment variables:

  BOARD="cm01/nrf9160/ns"
  GNUARMEMB_TOOLCHAIN_PATH="/usr"
  ZEPHYR_BASE="/home/martinv/Repos/ncs/zephyr"
  ZEPHYR_TOOLCHAIN_VARIANT="gnuarmemb"


-- Application: /home/martinv/Projekty/nextdrop/unified-fw_cm01test
-- CMake version: 3.30.7
-- Found Python3: /usr/bin/python (found suitable version "3.13.2", minimum required is "3.8") found components: Interpreter
-- Cache files will be written to: /home/martinv/.cache/zephyr
-- Zephyr version: 3.7.99 (/home/martinv/Repos/ncs/zephyr)
-- Found west (found suitable version "1.3.0", minimum required is "0.14.0")
-- Board: cm01, Revision: 1.0.0, qualifiers: nrf9160/ns
-- Found toolchain: gnuarmemb (/usr)
-- Found Dtc: /usr/bin/dtc (found suitable version "1.7.2", minimum required is "1.4.6")
-- Found BOARD.dts: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/zephyr/boards/nextdrop/cm01/cm01_nrf9160_ns.dts
-- Generated zephyr.dts: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/zephyr/zephyr.dts
-- Generated devicetree_generated.h: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/zephyr/include/generated/zephyr/devicetree_generated.h
-- Including generated dts.cmake file: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/zephyr/dts.cmake

warning: Deprecated symbol NET_SOCKETS_POSIX_NAMES is enabled.

Parsing /home/martinv/Repos/ncs/zephyr/Kconfig
Loaded configuration '/home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/zephyr/.config'
No change to configuration in '/home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/zephyr/.config'
No change to Kconfig header in '/home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/zephyr/include/generated/zephyr/autoconf.h'
-- Found GnuLd: /usr/arm-none-eabi/bin/ld.bfd (found version "2.43")
-- The C compiler identification is GNU 14.1.0
-- The CXX compiler identification is GNU 14.1.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/arm-none-eabi-gcc
=========== Generating psa_crypto_config ===============
Backup: CONFIG_MBEDTLS_PSA_CRYPTO_SPM: False
Backup: CONFIG_MBEDTLS_PSA_CRYPTO_C: False
Backup: CONFIG_MBEDTLS_PSA_CRYPTO_KEY_ID_ENCODES_OWNER: False
Backup: CONFIG_MBEDTLS_PLATFORM_SETUP_TEARDOWN_ALT: False
Backup: CONFIG_MBEDTLS_THREADING: False
Backup: CONFIG_MBEDTLS_THREADING_ALT: False
=========== Checkpoint: backup ===============
Restore: CONFIG_MBEDTLS_PSA_CRYPTO_SPM: False
Restore: CONFIG_MBEDTLS_PSA_CRYPTO_C: False
Restore: CONFIG_MBEDTLS_PSA_CRYPTO_KEY_ID_ENCODES_OWNER: False
Restore: CONFIG_MBEDTLS_PLATFORM_SETUP_TEARDOWN_ALT: False
Restore: CONFIG_MBEDTLS_THREADING: False
Restore: CONFIG_MBEDTLS_THREADING_ALT: False
=========== End psa_crypto_config ===============
=========== Generating psa_crypto_library_config ===============
Backup: CONFIG_MBEDTLS_PSA_CRYPTO_C: False
Backup: CONFIG_MBEDTLS_PSA_CRYPTO_KEY_ID_ENCODES_OWNER: False
Backup: CONFIG_MBEDTLS_PSA_CRYPTO_SPM: False
Backup: CONFIG_MBEDTLS_USE_PSA_CRYPTO: True
Backup: CONFIG_MBEDTLS_PLATFORM_PRINTF_ALT: False
Backup: CONFIG_MBEDTLS_THREADING: False
Backup: CONFIG_MBEDTLS_THREADING_ALT: False
=========== Checkpoint: backup ===============
Restore: CONFIG_MBEDTLS_PSA_CRYPTO_C: False
Restore: CONFIG_MBEDTLS_PSA_CRYPTO_KEY_ID_ENCODES_OWNER: False
Restore: CONFIG_MBEDTLS_PSA_CRYPTO_SPM: False
Restore: CONFIG_MBEDTLS_USE_PSA_CRYPTO: True
Restore: CONFIG_MBEDTLS_PLATFORM_PRINTF_ALT: False
Restore: CONFIG_MBEDTLS_THREADING: False
Restore: CONFIG_MBEDTLS_THREADING_ALT: False
=========== End psa_crypto_library_config ===============
-- Setting build type to 'MinSizeRel' as none was specified.
CMake Warning at /home/martinv/Repos/ncs/zephyr/CMakeLists.txt:2164 (message):
  

        The CMake build type was set to 'MinSizeRel', but the optimization flag was set to '-O2'.
        This may be intentional and the warning can be turned off by setting the CMake variable 'NO_BUILD_TYPE_WARNING'


CMake Deprecation Warning at /home/martinv/Repos/ncs/nrf/cmake/partition_manager.cmake:32 (message):
  

            ---------------------------------------------------------------------
            --- WARNING: Child and parent image functionality is deprecated   ---
            --- and should be replaced with sysbuild. Child and parent image  ---
            --- support remains only to allow existing customer applications  ---
            --- to build and allow porting to sysbuild, it is no longer       ---
            --- receiving updates or new features and it will not be possible ---
            --- to build using child/parent image at all in nRF Connect SDK   ---
            --- version 2.9 onwards.                                          ---
            ---------------------------------------------------------------------
Call Stack (most recent call first):
  /home/martinv/Repos/ncs/zephyr/cmake/modules/kernel.cmake:255 (include)
  /home/martinv/Repos/ncs/zephyr/cmake/modules/zephyr_default.cmake:142 (include)
  /home/martinv/Repos/ncs/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
  /home/martinv/Repos/ncs/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:92 (include_boilerplate)
  CMakeLists.txt:4 (find_package)


Dropping partition 'nrf_modem_lib_trace' since its size is 0.
Dropping partition 'nonsecure_storage' since it is empty.
-- Configuring done (5.8s)
-- Generating done (0.2s)
-- Build files have been written to: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01

[Finished]

Compiler log:

====================[ Build | merged_hex | cm-01 ]==============================
/usr/bin/cmake --build /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01 --target merged_hex -j 10
[0/30] Performing build step for 'tfm'
[68/72] Linking C executable bin/tfm_s.axf
Memory region         Used Size  Region Size  %age Used
           FLASH:       31996 B        32 KB     97.64%
             RAM:       14560 B        32 KB     44.43%
[72/72] Linking C static library secure_fw/libtfm_s_veneers.a
[3/26] Performing install step for 'tfm'
-- Install configuration: "RelWithDebInfo"
-- Installing: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/tfm/api_ns/bin/tfm_s.hex
-- Installing: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/tfm/api_ns/bin/tfm_s.bin
-- Installing: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/tfm/api_ns/bin/tfm_s.map
-- Installing: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/tfm/api_ns/bin/tfm_s.axf
-- Installing: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/tfm/api_ns/bin/tfm_s.elf
-- Installing: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/tfm/api_ns/interface/lib/s_veneers.o
-- Installing: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/tfm/api_ns/interface/include/nrf-config.h
-- Installing: /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/tfm/api_ns/interface/include/nrf-psa-crypto-config.h
----- Installing platform NS -----
[20/26] Linking CXX executable zephyr/zephyr_pre0.elf
/usr/lib/gcc/arm-none-eabi/14.1.0/../../../../arm-none-eabi/bin/ld.bfd: warning: zephyr/zephyr_pre0.elf has a LOAD segment with RWX permissions
[24/26] Linking CXX executable zephyr/zephyr.elf
/usr/lib/gcc/arm-none-eabi/14.1.0/../../../../arm-none-eabi/bin/ld.bfd: warning: zephyr/zephyr.elf has a LOAD segment with RWX permissions
Memory region         Used Size  Region Size  %age Used
           FLASH:       48772 B       992 KB      4.80%
             RAM:       13488 B     211608 B      6.37%
        IDT_LIST:           0 B        32 KB      0.00%
Generating files from /home/martinv/Projekty/nextdrop/unified-fw_cm01test/cmake-build-cm-01/zephyr/zephyr.elf for board: cm01
[26/26] Generating zephyr/merged.hex

Build finished

Application log:

All pins have been configured as non-secure
[1;34mBooting TF-M v2.1.1-ncs2-snapshot1[0m
[1;34m[Sec Thread] Secure image initializing![0m
TF-M isolation level is: 0x00000001
FATAL ERROR: SecureFault
Here is some context for the exception:
    EXC_RETURN (LR): 0xFFFFFFBD
    Exception came from non-secure FW in thread mode.
    xPSR:    0x20000007
    MSP:     0x20000BF8
    PSP:     0x20002838
    MSP_NS:  0x2000DCD8
    PSP_NS:  0x2000EDF0
    Exception frame at: 0x2000EDF0
        R0:   0x00000000
        R1:   0x00000000
        R2:   0x00000000
        R3:   0x00000000
        R12:  0x00000004
        LR:   0x00008C09
        PC:   0x00000000
        xPSR: 0x20000000
    Callee saved register state:        R4:   0x00000000
        R5:   0x00000000
        R6:   0x00000000
        R7:   0x00000000
        R8:   0x00000000
        R9:   0x00000000
        R10:  0x00000000
        R11:  0x00000000
    CFSR:  0x00000000
    BFSR:  0x00000000
    BFAR:  Not Valid
    MMFSR: 0x00000000
    MMFAR: Not Valid
    UFSR:  0x00000000
    HFSR:  0x00000000
    SFSR:  0x00000001
    SFAR: Not Valid

[Susheel Nuguru]

Just ran your project at my end and I see no assert. The main is running for a while in debugger and without debugger quite fine

[Susheel Nuguru]

wait, seems like you are running this in linux with tools outside the nrf connect sdk. 

I used windows with tools that came with nrf connect sdk v2.9.0

If you are testing this outside the toolbox that we recommend for our sdk, then I cannot test all those combinations. Please try to install our sdk and run your example with our toolchain that comes with it.

[Martin Vitek]

Yes, application builded with nrf sdk v2.9.0 is working fine. Unfortunately bundled GCC 12.2 is quite old for me, because I'm using new C++23/26 features.

I did some debugging:

- GCC 14.1 with binutils 2.43 from Fedora - working build, secure fault

- GCC 14.1 with binutils 2.44 from Fedora - build failing

[134/138] Linking C executable bin/tfm_s.axf
FAILED: bin/tfm_s.axf 
: && /usr/bin/arm-none-eabi-gcc -mcpu=cortex-m33+nofp -O2 -g -DNDEBUG -mcpu=cortex-m33+nofp --entry=Reset_Handler -specs=nano.specs -specs=nosys.specs -Wl,-check-sections -Wl,-fatal-warnings -Wl,--gc-sections -Wl,--no-wchar-size-warning -Wl,--print-memory-usage -Wl,-Map=/home/admin/Stažené/cm01test_2/build_fedora/tfm/bin/tfm_s.map -mfloat-abi=soft -Wl,--cmse-implib,--out-implib=/home/admin/Stažené/cm01test_2/build_fedora/tfm/secure_fw/s_veneers.o -T /home/admin/Stažené/cm01test_2/build_fedora/tfm/platform/target/nrf91/nrf_common/CMakeFiles/tfm_s_scatter.dir/./__/__/__/__/common/gcc/tfm_common_s.ld secure_fw/CMakeFiles/tfm_s.dir/partitions/ns_agent_tz/psa_api_veneers_v80m.o secure_fw/CMakeFiles/tfm_s.dir/__/platform/ext/target/nordic_nrf/common/core/startup.o secure_fw/CMakeFiles/tfm_s.dir/__/platform/ext/target/nordic_nrf/common/core/startup_nrf91.o secure_fw/CMakeFiles/tfm_s.dir/__/platform/ext/common/faults.o secure_fw/CMakeFiles/tfm_s.dir/__/generated/secure_fw/partitions/crypto/auto_generated/load_info_tfm_crypto.o secure_fw/CMakeFiles/tfm_s.dir/__/generated/secure_fw/partitions/platform/auto_generated/load_info_tfm_platform.o secure_fw/CMakeFiles/tfm_s.dir/partitions/ns_agent_tz/load_info_ns_agent_tz.o secure_fw/CMakeFiles/tfm_s.dir/__/platform/ext/common/syscalls_stub.o -o bin/tfm_s.axf  platform/libplatform_s.a  secure_fw/partitions/lib/runtime/libtfm_sprt.a  secure_fw/spm/libtfm_spm.a  secure_fw/partitions/crypto/libtfm_psa_rot_partition_crypto.a  platform/libplatform_crypto_keys.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/libcrypto_service_mbedcrypto.a  secure_fw/partitions/platform/libtfm_psa_rot_partition_platform.a  secure_fw/partitions/lib/runtime/libtfm_sp_log.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/libmbedcrypto_base.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/core/nrf_oberon/liboberon_psa_core.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/libnrf_security_utils.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/drivers/nrf_oberon/liboberon_psa_driver.a  /home/admin/Repos/ncs/v2.9.0/nrfxlib/crypto/nrf_oberon/lib/cortex-m33/soft-float/liboberon_mbedtls_3.0.15.a  platform/libplatform_s.a  secure_fw/partitions/lib/runtime/libtfm_sprt.a  secure_fw/spm/libtfm_spm.a  secure_fw/partitions/crypto/libtfm_psa_rot_partition_crypto.a  platform/libplatform_crypto_keys.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/libcrypto_service_mbedcrypto.a  secure_fw/partitions/platform/libtfm_psa_rot_partition_platform.a  secure_fw/partitions/lib/runtime/libtfm_sp_log.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/libmbedcrypto_base.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/core/nrf_oberon/liboberon_psa_core.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/libnrf_security_utils.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/drivers/nrf_oberon/liboberon_psa_driver.a  /home/admin/Repos/ncs/v2.9.0/nrfxlib/crypto/nrf_oberon/lib/cortex-m33/soft-float/liboberon_mbedtls_3.0.15.a  platform/libplatform_s.a  secure_fw/partitions/lib/runtime/libtfm_sprt.a  secure_fw/spm/libtfm_spm.a  secure_fw/partitions/crypto/libtfm_psa_rot_partition_crypto.a  platform/libplatform_crypto_keys.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/libcrypto_service_mbedcrypto.a  secure_fw/partitions/platform/libtfm_psa_rot_partition_platform.a  secure_fw/partitions/lib/runtime/libtfm_sp_log.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/libmbedcrypto_base.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/core/nrf_oberon/liboberon_psa_core.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/libnrf_security_utils.a  secure_fw/partitions/crypto/mbedcrypto/nrf_security_src/drivers/nrf_oberon/liboberon_psa_driver.a  /home/admin/Repos/ncs/v2.9.0/nrfxlib/crypto/nrf_oberon/lib/cortex-m33/soft-float/liboberon_mbedtls_3.0.15.a  /home/admin/Repos/ncs/v2.9.0/nrfxlib/crypto/nrf_cc310_platform/lib/cortex-m33/soft-float/no-interrupts/libnrf_cc310_platform_0.9.19.a  -lc  /home/admin/Repos/ncs/v2.9.0/nrfxlib/crypto/nrf_oberon/lib/cortex-m33/soft-float/liboberon_3.0.15.a && :
/usr/lib/gcc/arm-none-eabi/14.1.0/../../../../arm-none-eabi/bin/ld: /usr/lib/gcc/arm-none-eabi/14.1.0/thumb/v8-m.main/nofp/libgcc.a(cmse_nonsecure_call.o)(__gnu_cmse_nonsecure_call): Unknown destination type (ARM/Thumb) in platform/libplatform_s.a(ns_fault_service.o)
platform/libplatform_s.a(ns_fault_service.o): in function `call_ns_callback':
/home/admin/Repos/ncs/v2.9.0/nrf/modules/trusted-firmware-m/tfm_boards/common/ns_fault_service.c:139:(.text.call_ns_callback+0xc8): dangerous relocation: unsupported relocation
Memory region         Used Size  Region Size  %age Used
           FLASH:       31996 B        32 KB     97.64%
             RAM:       14560 B        32 KB     44.43%
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
FAILED: modules/trusted-firmware-m/tfm-prefix/src/tfm-stamp/tfm-build tfm/bin/tfm_s.elf tfm/bin/tfm_s.bin tfm/bin/tfm_s.hex tfm/bin/tfm_s_signed.bin tfm/bin/tfm_s_ns_signed.bin tfm/api_ns/interface/lib/s_veneers.o tfm/api_ns/interface/src/tfm_attest_api.c tfm/api_ns/interface/src/tfm_crypto_api.c tfm/api_ns/interface/src/tfm_fwu_api.c tfm/api_ns/interface/src/tfm_its_api.c tfm/api_ns/interface/src/tfm_platform_api.c tfm/api_ns/interface/src/tfm_ps_api.c tfm/api_ns/interface/src/tfm_tz_psa_ns_api.c tfm/api_ns/interface/src/tfm_ioctl_core_ns_api.c /home/admin/Stažené/cm01test_2/build_fedora/modules/trusted-firmware-m/tfm-prefix/src/tfm-stamp/tfm-build /home/admin/Stažené/cm01test_2/build_fedora/tfm/bin/tfm_s.elf /home/admin/Stažené/cm01test_2/build_fedora/tfm/bin/tfm_s.bin /home/admin/Stažené/cm01test_2/build_fedora/tfm/bin/tfm_s.hex /home/admin/Stažené/cm01test_2/build_fedora/tfm/bin/tfm_s_signed.bin /home/admin/Stažené/cm01test_2/build_fedora/tfm/bin/tfm_s_ns_signed.bin /home/admin/Stažené/cm01test_2/build_fedora/tfm/api_ns/interface/lib/s_veneers.o /home/admin/Stažené/cm01test_2/build_fedora/tfm/api_ns/interface/src/tfm_attest_api.c /home/admin/Stažené/cm01test_2/build_fedora/tfm/api_ns/interface/src/tfm_crypto_api.c /home/admin/Stažené/cm01test_2/build_fedora/tfm/api_ns/interface/src/tfm_fwu_api.c /home/admin/Stažené/cm01test_2/build_fedora/tfm/api_ns/interface/src/tfm_its_api.c /home/admin/Stažené/cm01test_2/build_fedora/tfm/api_ns/interface/src/tfm_platform_api.c /home/admin/Stažené/cm01test_2/build_fedora/tfm/api_ns/interface/src/tfm_ps_api.c /home/admin/Stažené/cm01test_2/build_fedora/tfm/api_ns/interface/src/tfm_tz_psa_ns_api.c /home/admin/Stažené/cm01test_2/build_fedora/tfm/api_ns/interface/src/tfm_ioctl_core_ns_api.c 
cd /home/admin/Stažené/cm01test_2/build_fedora/tfm && /usr/bin/cmake --build .
ninja: build stopped: subcommand failed.

- GCC 14.2.1 with binutils 2.43.1 from ARM - working build, secure fault

- GCC 13.3.1 with binutils 2.42 from ARM - working build, working app

- GCC 12.3.1 with binutils 2.40 from ARM - working build, working app

- GCC 14.2.1 with binutils 2.42 from ARM - working build, secure fault

For me now, using GCC 13.3 from ARM seems to be a solution.

[Susheel Nuguru]

Yes, unfortunately you might seem to need to compromise with 13.3. It seems like using later compiler versions might have exhausted the stack size of some of the thread or buffer which was already pretty close to being full. We will discover them when we move to newer version and most likely increase the stack sizes accordingly.