How to use ztest to calculate code coverage

[mtk0]

Hi

    How can I use ztest to calculate code coverage on the nrf54l15 platform, I tried to use the configurations of CONFIG_FORCE_COVERAGE=y, CONFIG_COVERAGE_GCOV=y, and CONFIG_COVERAGE=y to calculate the test coverage of ztest. However, after enabling these configurations, I encountered flash and RAM space overflow during compilation. The compilation log is as follows:

/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: region `FLASH' overflowed by 1541736 bytes
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: region `RAM' overflowed by 697999 bytes

[441/446] Linking C executable zephyr/zephyr_pre0.elf
FAILED: [code=1] zephyr/zephyr_pre0.elf zephyr/zephyr_pre0.map /data/mengkang1/work/x933/SE_P1_factory_0417/lc_zprojects/build/X933/zephyr/zephyr_pre0.map
: && ccache /opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc -gdwarf-4 zephyr/CMakeFiles/zephyr_pre0.dir/misc/empty_file.c.obj -o zephyr/zephyr_pre0.elf zephyr/CMakeFiles/offsets.dir/./arch/arm/core/offsets/offsets.c.obj -T zephyr/linker_zephyr_pre0.cmd -Wl,-Map=/data/mengkang1/work/x933/SE_P1_factory_0417/lc_zprojects/build/X933/zephyr/zephyr_pre0.map -Wl,--whole-archive app/libapp.a zephyr/libzephyr.a zephyr/arch/common/libarch__common.a zephyr/arch/arch/arm/core/libarch__arm__core.a zephyr/arch/arch/arm/core/cortex_m/libarch__arm__core__cortex_m.a zephyr/arch/arch/arm/core/cortex_m/cmse/libarch__arm__core__cortex_m__cmse.a zephyr/arch/arch/arm/core/mpu/libarch__arm__core__mpu.a zephyr/lib/libc/picolibc/liblib__libc__picolibc.a zephyr/lib/libc/common/liblib__libc__common.a zephyr/lib/net_buf/liblib__net_buf.a zephyr/soc/soc/nrf54l15/libsoc__nordic.a zephyr/subsys/debug/coredump/libsubsys__debug__coredump.a zephyr/subsys/fs/libsubsys__fs.a zephyr/subsys/mgmt/mcumgr/mgmt/libsubsys__mgmt__mcumgr__mgmt.a zephyr/subsys/mgmt/mcumgr/smp/libsubsys__mgmt__mcumgr__smp.a zephyr/subsys/mgmt/mcumgr/util/libsubsys__mgmt__mcumgr__util.a zephyr/subsys/mgmt/mcumgr/grp/img_mgmt/libsubsys__mgmt__mcumgr__grp__img_mgmt.a zephyr/subsys/mgmt/mcumgr/grp/os_mgmt/libsubsys__mgmt__mcumgr__grp__os_mgmt.a zephyr/subsys/mgmt/mcumgr/transport/libsubsys__mgmt__mcumgr__transport.a zephyr/subsys/random/libsubsys__random.a zephyr/subsys/testsuite/ztest/libsubsys__testsuite__ztest.a zephyr/subsys/bluetooth/common/libsubsys__bluetooth__common.a zephyr/subsys/bluetooth/host/libsubsys__bluetooth__host.a zephyr/subsys/dfu/boot/libsubsys__dfu__boot.a zephyr/subsys/zbus/libsubsys__zbus.a zephyr/drivers/adc/libdrivers__adc.a zephyr/drivers/cache/libdrivers__cache.a zephyr/drivers/clock_control/libdrivers__clock_control.a zephyr/drivers/console/libdrivers__console.a zephyr/drivers/entropy/libdrivers__entropy.a zephyr/drivers/flash/libdrivers__flash.a zephyr/drivers/gpio/libdrivers__gpio.a zephyr/drivers/hwinfo/libdrivers__hwinfo.a zephyr/drivers/i2c/libdrivers__i2c.a zephyr/drivers/pinctrl/libdrivers__pinctrl.a zephyr/drivers/rtc/libdrivers__rtc.a zephyr/drivers/serial/libdrivers__serial.a zephyr/drivers/spi/libdrivers__spi.a zephyr/drivers/timer/libdrivers__timer.a modules/nrf/lib/multithreading_lock/lib..__nrf__lib__multithreading_lock.a modules/nrf/subsys/bluetooth/controller/lib..__nrf__subsys__bluetooth__controller.a modules/nrf/subsys/nrf_security/src/zephyr/libmbedtls_zephyr.a modules/nrf/subsys/mpsl/init/lib..__nrf__subsys__mpsl__init.a modules/nrf/subsys/mpsl/fem/lib..__nrf__subsys__mpsl__fem.a modules/nrf/subsys/mpsl/hwres/lib..__nrf__subsys__mpsl__hwres.a modules/nrf/drivers/mpsl/clock_control/lib..__nrf__drivers__mpsl__clock_control.a modules/nrf/drivers/mpsl/flash_sync/lib..__nrf__drivers__mpsl__flash_sync.a modules/nrf/drivers/mpsl/temp_nrf5/lib..__nrf__drivers__mpsl__temp_nrf5.a modules/mcuboot/boot/bootutil/zephyr/libmcuboot_util.a modules/hal_nordic/modules/hal_nordic/nrfx/libmodules__hal_nordic__nrfx.a modules/littlefs/libmodules__littlefs.a modules/zcbor/libmodules__zcbor.a modules/drivers/onewire/YHM2010/lib..__lc_zprojects__drivers__onewire__YHM2010.a modules/drivers/aw2027/lib..__lc_zprojects__drivers__aw2027.a modules/drivers/ams/tcs3448/lib..__lc_zprojects__drivers__ams__tcs3448.a modules/drivers/semtech/sx9210/lib..__lc_zprojects__drivers__semtech__sx9210.a modules/drivers/st/lsm6dsv16x/lib..__lc_zprojects__drivers__st__lsm6dsv16x.a -Wl,--no-whole-archive zephyr/kernel/libkernel.a -L/data/mengkang1/work/x933/SE_P1_factory_0417/lc_zprojects/build/X933/zephyr modules/nrf/subsys/nrf_security/src/libmbedcrypto.a modules/nrf/subsys/nrf_security/src/core/nrf_oberon/libpsa_core.a modules/nrf/subsys/nrf_security/src/drivers/cracen/libcracen_psa_driver.a modules/nrf/subsys/nrf_security/src/libmbedcrypto_base.a modules/nrf/subsys/nrf_security/src/libnrf_security_utils.a zephyr/kernel/libkernel.a zephyr/arch/common/libisr_tables.a -mcpu=cortex-m33 -mthumb -mabi=aapcs -mfpu=fpv5-sp-d16 -mfloat-abi=hard -mfp16-format=ieee -mtp=soft -fuse-ld=bfd -Wl,--gc-sections -Wl,--build-id=none -Wl,--sort-common=descending -Wl,--sort-section=alignment -Wl,-u,_OffsetAbsSyms -Wl,-u,_ConfigAbsSyms -nostdlib -static -Wl,-X -Wl,-N -Wl,--orphan-handling=warn -Wl,-no-pie -specs=picolibc.specs -DPICOLIBC_DOUBLE_PRINTF_SCANF /data/mengkang1/work/x933/SE_P1_factory_0417/nrfxlib/softdevice_controller/lib/nrf54l/hard-float/libsoftdevice_controller_multirole.a /data/mengkang1/work/x933/SE_P1_factory_0417/nrfxlib/mpsl/fem/common/lib/nrf54l/hard-float/libmpsl_fem_common.a /data/mengkang1/work/x933/SE_P1_factory_0417/nrfxlib/mpsl/fem/nrf21540_gpio/lib/nrf54l/hard-float/libmpsl_fem_nrf21540_gpio.a /data/mengkang1/work/x933/SE_P1_factory_0417/nrfxlib/mpsl/fem/nrf21540_gpio_spi/lib/nrf54l/hard-float/libmpsl_fem_nrf21540_gpio_spi.a /data/mengkang1/work/x933/SE_P1_factory_0417/nrfxlib/mpsl/fem/nrf2220/lib/nrf54l/hard-float/libmpsl_fem_nrf2220.a /data/mengkang1/work/x933/SE_P1_factory_0417/nrfxlib/mpsl/fem/nrf2240/lib/nrf54l/hard-float/libmpsl_fem_nrf2240.a /data/mengkang1/work/x933/SE_P1_factory_0417/nrfxlib/mpsl/fem/nrf22xx/lib/nrf54l/hard-float/libmpsl_fem_nrf22xx.a /data/mengkang1/work/x933/SE_P1_factory_0417/nrfxlib/mpsl/fem/simple_gpio/lib/nrf54l/hard-float/libmpsl_fem_simple_gpio.a /data/mengkang1/work/x933/SE_P1_factory_0417/nrfxlib/mpsl/lib/nrf54l/hard-float/libmpsl.a -L"/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/thumb/v8-m.main+fp/hard" -lc -lgcc && cd /data/mengkang1/work/x933/SE_P1_factory_0417/lc_zprojects/build/X933/zephyr && /opt/cmake-4.0.2-linux-x86_64/bin/cmake -E true
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: zephyr/zephyr_pre0.elf section `text' will not fit in region `FLASH'
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: zephyr/zephyr_pre0.elf section `_GCOV_BSS_SECTION_NAME' will not fit in region `RAM'
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: region `FLASH' overflowed by 1541736 bytes
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: region `RAM' overflowed by 697999 bytes
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: modules/nrf/subsys/nrf_security/src/drivers/cracen/libcracen_psa_driver.a(mac.c.obj): in function `sx_mac_hw_reserve':
/data/mengkang1/work/x933/SE_P1_factory_0417/nrf/subsys/nrf_security/src/drivers/cracen/sxsymcrypt/src/mac.c:53: undefined reference to `sx_handle_nested_error'
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: modules/nrf/subsys/nrf_security/src/drivers/cracen/libcracen_psa_driver.a(mac.c.obj): in function `sx_mac_feed':
/data/mengkang1/work/x933/SE_P1_factory_0417/nrf/subsys/nrf_security/src/drivers/cracen/sxsymcrypt/src/mac.c:65: undefined reference to `sx_handle_nested_error'
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: /data/mengkang1/work/x933/SE_P1_factory_0417/nrf/subsys/nrf_security/src/drivers/cracen/sxsymcrypt/src/mac.c:68: undefined reference to `sx_handle_nested_error'
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: modules/nrf/subsys/nrf_security/src/drivers/cracen/libcracen_psa_driver.a(mac.c.obj): in function `sx_mac_run':
/data/mengkang1/work/x933/SE_P1_factory_0417/nrf/subsys/nrf_security/src/drivers/cracen/sxsymcrypt/src/mac.c:83: undefined reference to `sx_handle_nested_error'
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: modules/nrf/subsys/nrf_security/src/drivers/cracen/libcracen_psa_driver.a(mac.c.obj): in function `sx_mac_save_state':
/data/mengkang1/work/x933/SE_P1_factory_0417/nrf/subsys/nrf_security/src/drivers/cracen/sxsymcrypt/src/mac.c:152: undefined reference to `sx_handle_nested_error'
/opt/zephyr-sdk-0.17.0/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd: modules/nrf/subsys/nrf_security/src/drivers/cracen/libcracen_psa_driver.a(mac.c.obj):/data/mengkang1/work/x933/SE_P1_factory_0417/nrf/subsys/nrf_security/src/drivers/cracen/sxsymcrypt/src/mac.c:155: more undefined references to `sx_handle_nested_error' follow
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
FAILED: [code=1] _sysbuild/sysbuild/images/X933-prefix/src/X933-stamp/X933-build /data/mengkang1/work/x933/SE_P1_factory_0417/lc_zprojects/build/_sysbuild/sysbuild/images/X933-prefix/src/X933-stamp/X933-build
cd /data/mengkang1/work/x933/SE_P1_factory_0417/lc_zprojects/build/X933 && /opt/cmake-4.0.2-linux-x86_64/bin/cmake --build .
ninja: build stopped: subcommand failed.
FATAL ERROR: command exited with status 1: /usr/local/bin/cmake --build /data/mengkang1/work/x933/SE_P1_factory_0417/lc_zprojects/build
Error: ztest_case version build failed!

[Maria Gilje]

Hello, 

You would need to review your partitions and memory usage. Could you please share what your partitions are? Also, when you are not enabling the code coverage Kconfig symbols, what is the memory report at the end of your build log? 

Best regards,

Maria

[mtk0]

Hello

1, I am using the NRF54L15 chip, with 1.5MB of flash and 256KB of RAM.This is the partition table of my project,

app:
address: 0xd000
end_address: 0x8d000
region: flash_primary
size: 0x80000
mcuboot:
address: 0x0
end_address: 0xc000
placement:
align:
end: 0x1000
before:
- mcuboot_primary
region: flash_primary
size: 0xc000
mcuboot_pad:
address: 0xc000
end_address: 0xd000
placement:
align:
start: 0x1000
before:
- mcuboot_primary_app
region: flash_primary
size: 0x1000
mcuboot_primary:
address: 0xc000
end_address: 0x8d000
orig_span: &id001
- app
- mcuboot_pad
region: flash_primary
sharers: 0x1
size: 0x81000
span: *id001
mcuboot_primary_app:
address: 0xd000
end_address: 0x8d000
orig_span: &id002
- app
region: flash_primary
size: 0x80000
span: *id002
mcuboot_secondary:
address: 0x8d000
end_address: 0x10d000
placement:
after:
- mcuboot_primary
align:
start: 0x1000
region: flash_primary
share_size:
- mcuboot_primary_app
size: 0x80000
settings_storage:
address: 0x10d000
end_address: 0x10f000
placement:
after:
- app
align:
start: 0x1000
before:
- end
region: flash_primary
size: 0x2000
user_nvs:
address: 0x10f000
end_address: 0x111000
placement:
after:
- app
align:
start: 0x1000
before:
- end
region: flash_primary
size: 0x2000
settings_user_storage:
address: 0x111000
end_address: 0x16a000
placement:
after:
- app
align:
start: 0x1000
before:
- end
region: flash_primary
size: 0x59000
coredump_partition:
address: 0x16a000
end_address: 0x16e000
placement:
after:
- app
align:
start: 0x1000
before:
- end
region: flash_primary
size: 0x4000
offline_logging:
address: 0x16e000
end_address: 0x17c000
placement:
after:
- app
align:
start: 0x1000
before:
- end
region: flash_primary
size: 0xe000
dtm_flag:
address: 0x17c000
end_address: 0x17d000
placement:
after:
- app
align:
start: 0x1000
before:
- end
region: flash_primary
size: 0x1000
otp:
address: 0xffd500
end_address: 0xffd9fc
region: otp
size: 0x4fc
sram_primary:
address: 0x20000000
end_address: 0x2002f000
region: sram_primary
size: 0x2f000

2,  This is the compilation log when closing VNet FORCE-COVERAGE=n, VNet COVERAGE_GCOV=n, and VNet COVERAGE=n:

[Maria Gilje]

Hello, 

Thank you for the additional information. Since your partitions indicate that you are using dual bank DFU, this overflow is not very surprising.

mtk0 said:

2,  This is the compilation log when closing VNet FORCE-COVERAGE=n, VNet COVERAGE_GCOV=n, and VNet COVERAGE=n:

Your application is already at 85% of the allocated flash before enabling the coverage symbols. Would it have any value for you to do a coverage test without the DFU configurations? It could at least shed some light on how much the memory usage increases when enabling the coverage symbols. If it seems feasible with those numbers that you could be able to fine tune your partitions to barely fit the application with the coverage symbols enabled, then I'd say you should give it a try. But do please note that we don't recommend shipping products with 100% flash usage because it can make it impossible to perform future upgrades. 

Also, to improve readability of your latest reply, I will edit the formatting of it and put the partition code into a code block (with Insert->Code). Please use Insert->Code for logs and code in the future, thanks!

Best regards,

Maria

[mtk0]

Hi

This issue is not about configuring flash partitions to allow enough space for firmware to flow out. I have also tried using this routine nrf/samples/blue/peripherals_uart to add CONFIG_FORCE_COVERAGE=y, CONFIG_COVERAGE_GCOV=y, and CONFIG_COVERAGE=y in prj.conf, which will also report the same error during compilation. Therefore, I believe that using these three configurations is not suitable for the nrf54l15 chip. What is the correct way to test code coverage on nrf54l15 using ztest.

The following is the log compiled from prj.conf in nrf/samples/blue/peripheral-uart with the addition of CONFIG_FORCE_COVERAGE=y, CONFIG_COVERAGE_GCOV=y, and CONFIG_COVERAGE=y

[Maria Gilje]

Hello,

Yes, I see this now. I see the same when adding the coverage symbols to Zephyr samples. So the RAM and NVM requirements that coverage has is too large to run on the nRF54L15. Instead of running on the hardware, you could check out using Twister instead to generate coverage reports. 

Best regards,

Maria

[mtk0]

Thank you very much for your answer. I will first learn how to use Twitter, and then use Twitter to calculate our code testing coverage.