Enabling coverage reports for device testing with nRF Connect SDK

[Jason Wright]

This may just be more of a Zephyr question but wondering if anyone in the Nordic community has experience with this.

I'd like to generate code coverage reports when running tests using the Zephyr Test Framework (Ztest). This works well when testing on the native_posix platform, but when running on actual hardware e.g.:

twister -T . -j1 --device-testing --device-serial /dev/ttyACM2 --device-serial-baud 115200 -p nrf5340dk_nrf5340_cpuapp --coverage

Coverage is disabled due to unsatisfied dependency HAS_COVERAGE_SUPPORT:

warning: COVERAGE (defined at subsys/testsuite/Kconfig:33) was assigned the value 'y' but got the

value 'n'. Check these unsatisfied dependencies: HAS_COVERAGE_SUPPORT (=n). See

docs.zephyrproject.org/.../kconfig.html and/or look up COVERAGE in the

menuconfig/guiconfig interface. The Application Development Primer, Setting Configuration Values,

and Kconfig - Tips and Best Practices sections of the manual might be helpful too.

The description of HAS_COVERAGE_SUPPORT says:

The code coverage report generation is only available on boards
with enough spare RAM to buffer the coverage data, or on boards
based on the POSIX ARCH.

I tested changing HAS_COVERAGE_SUPPORT in Kconfig.board, but this caused the test to timeout. Is it possible to get this working?

[dejans]

Hi,

I have reported the issue internally.
I'll get back to you when I get new information, hopefully till the end of next week.

Best regards,
Dejan

[dejans]

Hi,

I am sorry for the delay. Due to vacation time, you may need to wait for the reply longer than initially expected. I'll come back to you when I get new information, most probably in a week or two after August 15.

Best regards,
Dejan

[dejans]

Hi,

I have found the discussion related to your question on Zephyr's Discord #nordic channel. As mentioned there, the main issue is related to the RAM requirement.

Is the reply that you got on Discord channel satisfactory?

Best regards,
Dejan

[Jason Wright]

Hi Dejan,

Not quite, because although CONFIG_CPLUSPLUS provides a workaround for console output halting, I still could not get the coverage report.

However, digging on this further today, I discovered the heap size for coverage data is hardcoded to 16384 bytes in Zephyr: https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/testsuite/coverage/coverage.c#L18

And occasionally that would be not enough to allocate the gcov buffer, resulting in "No Mem available to continue dump" being printed in the gcov files which breaks parsing.

By manually increasing MALLOC_MAX_HEAP_SIZE to 32768 bytes I was able to generate coverage reports for the attached sample. I also needed to increase CONFIG_MAIN_STACK_SIZE in prj.conf. So this issue can be closed, I will pursue a cleaner way of changing the coverage heap size with Zephyr team.

Example attached, I was able to generate the coverage report on nrf5340dk by running 
twister -T . --device-testing --device-serial /dev/ttyACM2 --device-ser
ial-baud 115200 -p nrf5340dk_nrf5340_cpuapp --coverage

hello_world_coverage.zip

[dejans]

Hi,

Thank you for providing your findings.

I am really glad that you found the way to get the coverage report.

Jason Wright said:

So this issue can be closed, I will pursue a cleaner way of changing the coverage heap size with Zephyr team.

If you want, you can close the case now.

Best regards,
Dejan