Enabling coverage reports for device testing with nRF Connect SDK

RE: Enabling coverage reports for device testing with nRF Connect SDK

dejans — Fri, 19 Aug 2022 07:33:23 GMT

Hi,

Thank you for providing your findings.

I am really glad that you found the way to get the coverage report.

[quote user="jpwright0"] So this issue can be closed, I will pursue a cleaner way of changing the coverage heap size with Zephyr team.[/quote]

If you want, you can close the case now.

Best regards,
Dejan

RE: Enabling coverage reports for device testing with nRF Connect SDK

Jason Wright — Thu, 18 Aug 2022 15:37:25 GMT

Hi Dejan,

Not quite, because although CONFIG_CPLUSPLUS provides a workaround for console output halting, I still could not get the coverage report.

However, digging on this further today, I discovered the heap size for coverage data is hardcoded to 16384 bytes in Zephyr: https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/testsuite/coverage/coverage.c#L18

And occasionally that would be not enough to allocate the gcov buffer, resulting in "No Mem available to continue dump" being printed in the gcov files which breaks parsing.

By manually increasing MALLOC_MAX_HEAP_SIZE to 32768 bytes I was able to generate coverage reports for the attached sample. I also needed to increase CONFIG_MAIN_STACK_SIZE in prj.conf. So this issue can be closed, I will pursue a cleaner way of changing the coverage heap size with Zephyr team.

Example attached, I was able to generate the coverage report on nrf5340dk by running
twister -T . --device-testing --device-serial /dev/ttyACM2 --device-ser
ial-baud 115200 -p nrf5340dk_nrf5340_cpuapp --coverage

devzone.nordicsemi.com/.../hello_5F00_world_5F00_coverage.zip

RE: Enabling coverage reports for device testing with nRF Connect SDK

dejans — Thu, 18 Aug 2022 12:00:04 GMT

Hi,

I have found the discussion related to your question on Zephyr's Discord #nordic channel. As mentioned there, the main issue is related to the RAM requirement.

Is the reply that you got on Discord channel satisfactory?

Best regards,
Dejan

RE: Enabling coverage reports for device testing with nRF Connect SDK

dejans — Mon, 08 Aug 2022 07:56:28 GMT

Hi,

I am sorry for the delay. Due to vacation time, you may need to wait for the reply longer than initially expected. I'll come back to you when I get new information, most probably in a week or two after August 15.

Best regards,
Dejan

RE: Enabling coverage reports for device testing with nRF Connect SDK

dejans — Wed, 27 Jul 2022 15:50:55 GMT

Hi,

I have reported the issue internally.
I'll get back to you when I get new information, hopefully till the end of next week.

Best regards,
Dejan

RE: Enabling coverage reports for device testing with nRF Connect SDK

dejans — Tue, 26 Jul 2022 16:39:06 GMT

Hi,

I have reproduced the issue. I would need to investigate this further.
I'll come back to you with new information, hopefully till the end of this week.

Best regards,
Dejan

RE: Enabling coverage reports for device testing with nRF Connect SDK

Jason Wright — Mon, 25 Jul 2022 15:35:17 GMT

Yes, that's right - nothing changed in configuration except enabling coverage. Here are the exact steps using the Zephyr hello_world example:

(1) Running the Zephyr hello_world example unmodified:

*** Booting Zephyr OS build v3.0.99-ncs1 ***
Hello World! nrf5340dk_nrf5340_cpuapp

(2) Running with coverage support enabled

- add "select HAS_COVERAGE_SUPPORT" to ncs/zephyr/boards/arm/nrf5340dk_nrf5340/Kconfig.board line 10
- add CONFIG_COVERAGE=y to prj.conf

no terminal output

(3) Based on advice from someone on Zephyr discord, I also tried running with C++ support (related to this issue: https://github.com/zephyrproject-rtos/zephyr/issues/40668)

- add CONFIG_CPLUSPLUS=y to prj.conf

*** Booting Zephyr OS build prints repeatedly, looks like CPU is resetting due to a hardfault, but at least get terminal output

RE: Enabling coverage reports for device testing with nRF Connect SDK

dejans — Mon, 25 Jul 2022 15:15:04 GMT

Hi,

When coverage support is disabled you can see the expected output in the terminal application, but when coverage is enabled, there is nothing displayed in the terminal even though you have not changed anything in the configuration. Is this correct?

Can you provide more information on what you can see in the terminal application connected to the serial port when coverage support is not enabled?

Best regards,
Dejan

RE: Enabling coverage reports for device testing with nRF Connect SDK

Jason Wright — Fri, 22 Jul 2022 20:06:13 GMT

Thank you - This is not really a Twister issue, and I was able to build with coverage support by selecting HAS_COVERAGE_SUPPORT in Kconfig.board. However, tests do not seem to run properly when COVERAGE is enabled. There is no output on the serial port. Other boards like mps2_an385 support this so it should be possible, if the nRF5340 has enough RAM, but how much RAM is sufficient?
Having coverage reports is important for my application.

RE: Enabling coverage reports for device testing with nRF Connect SDK

dejans — Wed, 20 Jul 2022 13:40:14 GMT

Hi,

CONFIG_HAS_COVERAGE_SUPPORT should be selected in Kconfig.board file to enable device coverage. Reporting the coverage for any particular application also requires CONFIG_COVERAGE. You can take a look at generating coverage reports. There you can find a link to gcov test coverage program. Note that you need to have sufficient RAM available when enabling coverage for the device.

You can also take a look at the cryptography tests sample which uses Ztest to run tests.

As Twister is Zephyr test script, there are several places where you can ask Zephyr related questions.

Best regards,
Dejan