nrf9160 ztest framework : unable to get a unit test to work

[emeralddragon]

Hi. I am working on the nrf9160. I want to use a unit test framework for my application code (which is a combination of C/C++14). I thought about using gtest/gmock or ztest. For now I am trying to get to run a single sample provided: zephyr/sample/testing/integration up and running but unable to do so. I am working on a win 64 laptop.  Can you please explain how I can run a unit test please. I looked at the following webpages:https://docs.zephyrproject.org/1.12.0/subsystems/test/ztest.html, https://docs.zephyrproject.org/latest/reference/kconfig/CONFIG_ZTEST.html, https://developer.nordicsemi.com/nRF_Connect_SDK/doc/0.4.0/zephyr/guides/test/ztest.html. Please dont direct me to the same webpages.

what are the steps to run a unit/integration test on the win 64 enviroment? Do I have to run cmake? Trying to run the sanitycheck script fails due to several reasons:

On cmd: 

xxxxx\zephyr\scripts>python sanitycheck -s ..\samples\testing\integration
Install the anytree module to use the --test-tree option
Install tabulate python module with pip to use --device-testing option.
Renaming output directory to xxx\zephyr\scripts\sanity-out.4
INFO - JOBS: 8
INFO - Selecting default platforms per test case
INFO - Building initial testcase list...
INFO - 0 test configurations selected, 163531 configurations discarded due to filters.
INFO - Adding tasks to the queue...

INFO - [32m0 of 0[39m tests passed (0.00%), [39m0[39m failed, 0 skipped with [39m0[39m warnings in 7.89 seconds
INFO - In total 741 test cases were executed on 0 out of total 225 platforms (0.00%)
'stty' is not recognized as an internal or external command,
operable program or batch file.

On Cygwin: "/usr/bin/env: ‘python3’: No such file or directory" python3 not found. I had to change the shebang in sanitycheck to python instead. After that another error:

$ xxx/zephyr/scripts/sanitycheck -s ../samples/testing/integration
Install the anytree module to use the --test-tree option
Install tabulate python module with pip to use --device-testing option.
Traceback (most recent call last):
File "xxx/zephyr/scripts/sanitycheck", line 213, in <module>
import edtlib
ModuleNotFoundError: No module named 'edtlib'

when I try to source the zephyr env variable on cygwin: here is the error i get: 

source xxx/zephyr/zephyr-env.sh
-bash: $'\r': command not found
-bash: xxx/zephyr/zephyr-env.sh: line 70: syntax error: unexpected end of file

All I want to do is write unit tests and then eventually integration tests using the ztest framework, first on win64 platform. I hope someone can help me out. Please provide a step wise approach with commands to be run. Noob here.

[Simon]

Could you test it again in Cygwin and run the following commands first:

cd <sourcecode_root>\ncs
pip3 install -r zephyr\scripts\requirements.txt
pip3 install -r nrf\scripts\requirements.txt
pip3 install -r bootloader\mcuboot\scripts\requirements.txt

Best regards,

Simon

[emeralddragon]

Thanks Simon. I did as you said,  anytree, tabulate were not yet installed. But the issue remains: 

zephyr/scripts/sanitycheck -s ../samples/testing/integration
Traceback (most recent call last):
File "zephyr/scripts/sanitycheck", line 213, in <module>
import edtlib
ModuleNotFoundError: No module named 'edtlib'

[emeralddragon]

So paths are an issue. It is unable to find the zephyr/scripts/dts path to find the edts/dts files. Any recommendations on how to fix this? I do not want to edit the zephyr source files.

[Simon]

I got the same error message when trying to build it in windows cmd. However, I was able to make it work using VirtualBox Ubuntu, but I'll set up Cygwin and see if I am able to make it work there as well. I'll keep you updated.

Best regards,

Simon

[emeralddragon]

Okay. Just to mention: VB Ubuntu is not a preferred option...as of now.

[Simon]

It seems like this line is the one that includes the path where zephyr/scripts/dts/edtlib.py is located.

Have you set the ZEPHYR_BASE?

Could you try to add the following below this line (above import edtlib), to see what the path is equal to:

print("scripts/dts/ path",os.path.join(ZEPHYR_BASE, "scripts", "dts"))

Best regards,

Simon

[emeralddragon]

Hi SImon,

I see what I got previously after adding "dts." to the import paths.

$ zephyr/scripts/sanitycheck -s ../samples/testing/integration

scripts/dts/ path C:\ConnectivityNodes_Cellular\zephyr\scripts\dts
Traceback (most recent call last):
File "zephyr/scripts/sanitycheck", line 1403, in <module>
class Platform:
File "zephyr/scripts/sanitycheck", line 1408, in Platform
platform_schema = scl.yaml_load(os.path.join(ZEPHYR_BASE,
File "C:\ConnectivityNode_Cellular\zephyr\scripts\sanity_chk\scl.py", line 29, in yaml_load
with open(filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\ConnectivityNodes_Cellular\\zephyr\\scripts\\sanity_chk\\platform-schema.yaml'

[Simon]

I managed to make this work, I did the following:

• Installed ncs v1.2.0. Check out our newly released app Toolchain Manager, which simplifies the process of installing ncs

• Opened Git Bash
• Typed in:

cd <location_of_ncs>/ncs/zephyr
export ZEPHYR_BASE=c:/Nordic_SDK/ncs/zephyr

• Copied the sample ..\ncs\zephyr\samples\testing\integration into ..\ncs\zephyr\tests
• Ran the following command:

python scripts/sanitycheck -s tests/integration

• Got the following output:

INFO    - JOBS: 8
INFO    - Selecting default platforms per test case
INFO    - Building initial testcase list...
INFO    - 0 test configurations selected, 163755 configurations discarded due to filters.
INFO    - Adding tasks to the queue...
INFO    - 0 of 0 tests passed (0.00%), 0 failed, 0 skipped with 0 warnings in 4.73 seconds
INFO    - In total 742 test cases were executed on 0 out of total 225 platforms (0.00%)
Renaming output directory to C:\Nordic_SDK\ncs\zephyr\sanity-out.4

Best regards,

Simon

[emeralddragon]

Hi Simon. Thanks for the update. After copying the folder to zephyr/tests it seems to be working. Here is my output:

INFO - JOBS: 8
INFO - Selecting default platforms per test case
INFO - Building initial testcase list...
INFO - 0 test configurations selected, 163755 configurations discarded due to filters.
INFO - Adding tasks to the queue...
INFO - 0 of 0 tests passed (0.00%), 0 failed, 0 skipped with 0 warnings in 5.14 seconds
INFO - In total 742 test cases were executed on 0 out of total 225 platforms (0.00%)
scripts/dts/ path xxx/zephyr\scripts\dts
Renaming output directory to xxx\zephyr\sanity-out.1

But I am surprised that the script found 742 tests and it was not run on any platform. It says  "0 test configurations selected" and "0 of 0 tests passed (0.00%), 0 failed" How do I run the testcase from main.c and see the output?

[Vojislav]

Hi,

I am also trying to get unit testing working on nrf91, and also got to this part. I am developing on Ubuntu. What are the next steps to get tests from main.c running?

Best regards,
Vojislav

[Simon]

Sorry for the delay, I've been gone due to easter holidays.

Could you open a new ticket and ask your question there, since the questions are somewhat unrelated(The first question was about getting tests to work on windows and this question is asking specifically about the tests)? This way we can keep everything more organized. 

Best regards,

Simon

[emeralddragon]

Hi Simon, Not a problem. As the heading says: unable to get a unit test to work. I don't see the test in the main.c being run, and my questions previously still hold.Here is a clear list of questions:

• The sanitycheck script did run without errors on windows but the output of that script needs some explanations. What are the 742 tests found? Why are 0 tests run etc. Please help me understand these. 
• Why can I not see the tests from main.c ? What am I missing?