Enabling coverage reports for device testing with nRF Connect SDK

[Jason Wright]

This may just be more of a Zephyr question but wondering if anyone in the Nordic community has experience with this.

I'd like to generate code coverage reports when running tests using the Zephyr Test Framework (Ztest). This works well when testing on the native_posix platform, but when running on actual hardware e.g.:

twister -T . -j1 --device-testing --device-serial /dev/ttyACM2 --device-serial-baud 115200 -p nrf5340dk_nrf5340_cpuapp --coverage

Coverage is disabled due to unsatisfied dependency HAS_COVERAGE_SUPPORT:

warning: COVERAGE (defined at subsys/testsuite/Kconfig:33) was assigned the value 'y' but got the

value 'n'. Check these unsatisfied dependencies: HAS_COVERAGE_SUPPORT (=n). See

docs.zephyrproject.org/.../kconfig.html and/or look up COVERAGE in the

menuconfig/guiconfig interface. The Application Development Primer, Setting Configuration Values,

and Kconfig - Tips and Best Practices sections of the manual might be helpful too.

The description of HAS_COVERAGE_SUPPORT says:

The code coverage report generation is only available on boards
with enough spare RAM to buffer the coverage data, or on boards
based on the POSIX ARCH.

I tested changing HAS_COVERAGE_SUPPORT in Kconfig.board, but this caused the test to timeout. Is it possible to get this working?

[dejans]

Hi,

CONFIG_HAS_COVERAGE_SUPPORT should be selected in Kconfig.board file to enable device coverage. Reporting the coverage for any particular application also requires CONFIG_COVERAGE. You can take a look at generating coverage reports. There you can find a link to gcov test coverage program. Note that you need to have sufficient RAM available when enabling coverage for the device.

You can also take a look at the cryptography tests sample which uses Ztest to run tests.

As Twister is Zephyr test script, there are several places where you can ask Zephyr related questions.

• GitHub issue 
• Zephyr Discord channel 
• Mailing list

Best regards,
Dejan

[Jason Wright]

Thank you - This is not really a Twister issue, and I was able to build with coverage support by selecting HAS_COVERAGE_SUPPORT in Kconfig.board. However, tests do not seem to run properly when COVERAGE is enabled. There is no output on the serial port. Other boards like mps2_an385 support this so it should be possible, if the nRF5340 has enough RAM, but how much RAM is sufficient?
Having coverage reports is important for my application.

[dejans]

Hi,

When coverage support is disabled you can see the expected output in the terminal application, but when coverage is enabled, there is nothing displayed in the terminal even though you have not changed anything in the configuration. Is this correct?

Can you provide more information on what you can see in the terminal application connected to the serial port when coverage support is not enabled?

Best regards,
Dejan

[Jason Wright]

Yes, that's right - nothing changed in configuration except enabling coverage. Here are the exact steps using the Zephyr hello_world example:

(1) Running the Zephyr hello_world example unmodified:

*** Booting Zephyr OS build v3.0.99-ncs1  ***
Hello World! nrf5340dk_nrf5340_cpuapp

(2) Running with coverage support enabled

- add "select HAS_COVERAGE_SUPPORT" to ncs/zephyr/boards/arm/nrf5340dk_nrf5340/Kconfig.board line 10
- add CONFIG_COVERAGE=y to prj.conf

no terminal output

(3) Based on advice from someone on Zephyr discord, I also tried running with C++ support (related to this issue: https://github.com/zephyrproject-rtos/zephyr/issues/40668)

- add CONFIG_CPLUSPLUS=y to prj.conf

*** Booting Zephyr OS build prints repeatedly, looks like CPU is resetting due to a hardfault, but at least get terminal output

[dejans]

Hi,

I have reproduced the issue. I would need to investigate this further.
I'll come back to you with new information, hopefully till the end of this week.

Best regards,
Dejan

[dejans]

Hi,

I have reported the issue internally.
I'll get back to you when I get new information, hopefully till the end of next week.

Best regards,
Dejan

[dejans]

Hi,

I am sorry for the delay. Due to vacation time, you may need to wait for the reply longer than initially expected. I'll come back to you when I get new information, most probably in a week or two after August 15.

Best regards,
Dejan

[dejans]

Hi,

I have found the discussion related to your question on Zephyr's Discord #nordic channel. As mentioned there, the main issue is related to the RAM requirement.

Is the reply that you got on Discord channel satisfactory?

Best regards,
Dejan

[Jason Wright]

Hi Dejan,

Not quite, because although CONFIG_CPLUSPLUS provides a workaround for console output halting, I still could not get the coverage report.

However, digging on this further today, I discovered the heap size for coverage data is hardcoded to 16384 bytes in Zephyr: https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/testsuite/coverage/coverage.c#L18

And occasionally that would be not enough to allocate the gcov buffer, resulting in "No Mem available to continue dump" being printed in the gcov files which breaks parsing.

By manually increasing MALLOC_MAX_HEAP_SIZE to 32768 bytes I was able to generate coverage reports for the attached sample. I also needed to increase CONFIG_MAIN_STACK_SIZE in prj.conf. So this issue can be closed, I will pursue a cleaner way of changing the coverage heap size with Zephyr team.

Example attached, I was able to generate the coverage report on nrf5340dk by running 
twister -T . --device-testing --device-serial /dev/ttyACM2 --device-ser
ial-baud 115200 -p nrf5340dk_nrf5340_cpuapp --coverage

hello_world_coverage.zip