Unable to Run ZTest Unit Tests on nRF Sample – native_sim Not Detected by Twister

[SemihAydin]

Hello everyone,

I would like to learn how to create unit tests using the ZTest framework for a Nordic sample project, in order to better understand how it works.

I am currently working on a Linux (Ubuntu) machine, as I understand from the documentation that this is the recommended approach for running unit tests without hardware.

Since I couldn’t find any example that directly applies ZTest to a Nordic sample, I decided to try creating my own test setup.
Here's what I’ve done so far:

1. I created a new workspace and added the nrf_cloud_multi_service sample.

2. I then attempted to create a unit test, which you can find at tests/unit directory in the zip file:
   
    multi-main.zip

3. To run the test, I used the following command: python3 /home/sirdarwin/ncs/v3.0.0/zephyr/scripts/twister -T /home/sirdarwin/nrf_multi/tests/unit -p native_sim -v

However, the test did not run as expected. I’m attaching the terminal output, along with the build.log and CMakeOutput.log files for your review.

INFO    - Using Ninja..
INFO    - Zephyr version: v4.0.99-ncs1
INFO    - Using 'zephyr' toolchain.
INFO    - Building initial testsuite list...
INFO    - Writing JSON report /home/sirdarwin/nrf_multi/tests/unit/twister-out/testplan.json
INFO    - JOBS: 8
INFO    - Adding tasks to the queue...
INFO    - Added initial list of jobs to queue
ERROR   - CMake build failure: /home/sirdarwin/nrf_multi/tests/unit for native_sim/native
INFO    - 1/1 native_sim/native         nrf.cloud.loc_test                                 ERROR CMake build failure (native <host>)
ERROR   - see: /home/sirdarwin/nrf_multi/tests/unit/twister-out/native_sim_native/host/nrf.cloud.loc_test/build.log

INFO    - 1 test scenarios (1 configurations) selected, 0 configurations filtered (0 by static filter, 0 at runtime).
INFO    - 0 of 1 executed test configurations passed (0.00%), 0 built (not run), 0 failed, 1 errored, with no warnings in 4.76 seconds.
INFO    - 0 of 3 executed test cases passed (0.00%), 3 errored on 1 out of total 929 platforms (0.11%).
INFO    - 0 test configurations executed on platforms, 1 test configurations were only built.
INFO    - Saving reports...
INFO    - Writing JSON report /home/sirdarwin/nrf_multi/tests/unit/twister-out/twister.json
INFO    - Writing xunit report /home/sirdarwin/nrf_multi/tests/unit/twister-out/twister.xml...
INFO    - Writing xunit report /home/sirdarwin/nrf_multi/tests/unit/twister-out/twister_report.xml...
INFO    - -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
INFO    - The following issues were found (showing the top 10 items):
INFO    - 1) nrf.cloud.loc_test on native_sim/native error (CMake build failure)
INFO    -
INFO    - To rerun the tests, call twister using the following commandline:
INFO    - west twister -p <PLATFORM> -s <TEST ID>, for example:
INFO    -
INFO    - west twister -p native_sim/native -s nrf.cloud.loc_test
INFO    - or with west:
INFO    - west build -p -b native_sim/native . -T nrf.cloud.loc_test
INFO    - -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
INFO    - Run completed

Please note that the repository I shared is identical to the nrf_multi project for which I attached the logs. Just in case the issue was related to using a cloned repository, I also created the same project (nrf_cloud_multi_service_sample) from the nRF Connect SDK using VS Code and named it nrf_multi. I then manually added the same unit test files to this local project as well.

I’ve verified that my ZEPHYR_BASE is correctly set by running: echo $ZEPHYR_BASE
which outputs: /home/sirdarwin/ncs/v3.0.0/zephyr

The main issue I’m facing is that Twister doesn’t seem to detect the native_sim or native_posix platform for executing the test. Error as follows:

No board named 'native_sim' found.

Please choose one of the following boards:

unit_testing
nrf54lv10apdk
nrf7120pdk
nrf52dmouse
nrf52kbd
thingy91x
nrf52840gmouse
nrf54lm20apdk
nrf52833dongle
nrf52820dongle
thingy91
CMake Error at /home/sirdarwin/ncs/v3.0.0/zephyr/cmake/modules/boards.cmake:228 (message):
  Invalid BOARD; see above.

Please note that I’m relatively new to both Linux and the Zephyr RTOS environment. I’m trying to learn by doing, and I believe that writing tests is a great way to deepen my understanding.

There’s a high chance I may have made some mistakes in the process, and I’d really appreciate it if someone could review my steps and guide me on the correct approach to set up and run unit tests using ZTest.

Thank you for your time and support!

Best regards,
Semih

[dejans]

Hi Semih,

Are you sure that you have and use correct Toolchain?

It could also be worth double-checking that you have set correct ZEPHYR_BASE because in you case it seems that twister does not see boards from the Zephyr's root.

Best regards,
Dejan

[SemihAydin]

Hi Dejans,

Thank you so much for your reply.

I'm sure that I'm using same toolchain(v3.0.0) but there are two toolchain installed other one is v2.9.0

As I mentioned that I'm checking my ZEPHYR_BASE by running: echo $ZEPHYR_BASE on terminal and I'm getting
"/home/sirdarwin/ncs/v3.0.0/zephyr" at the terminal output.

I will try to uninstall other toolchain then I will keep you updated.

Best regards,
Semih

[dejans]

Hi Semih,

SemihAydin said:

I will try to uninstall other toolchain then I will keep you updated.

Thank you. I am waiting for your update.

As it seems that twister cannot find boards from Zephyr root, you could try to add "--board-root" to your twister command. You can use "west twister --help" to get more information.

Best regards,
Dejan

[SemihAydin]

Hİ Dejan,

I uninstalled all folders belongs to zephyr and nRF like ncs, nrf connect from vs code and zephyr folders from computer, then I installed them again I also removed the zephyr's path from the /.bashrc. After this process, result wasn't different. However, I decided to open some space in my computer because it was around 100-200MB space left. I opened more than 5GB space and this time result was different, I didn't get any error with native_sim.

This is good part, but now I have a different question.

I copied the intmath test from https://github.com/nrfconnect/sdk-zephyr/tree/v4.0.99-ncs1/tests/unit to see if my tests are wrong.

Here are the results,

native_sim and native_posix: I just added the necessary platform_allow and harness informations to the testcase.yaml file and tried to run with it. There are no error but tests are not executed. If I add build_only: false or build_only: true option, results are the same with native platforms.
    platform_allow:
      - native_sim
      - native_posix
    harness: ztest

My run commands are:
python3 /home/sirdarwin/ncs/v3.0.0/zephyr/scripts/twister -p native_posix -T /home/sirdarwin/multi_service/tests/unit/intmath -v

python3 /home/sirdarwin/ncs/v3.0.0/zephyr/scripts/twister -p native_sim -T /home/sirdarwin/multi_service/tests/unit/intmath -v



unit_testing: This platform is working as expected, I run the command "python3 /home/sirdarwin/ncs/v3.0.0/zephyr/scripts/twister -p unit_testing -T /home/sirdarwin/multi_service/tests/unit/intmath -v" and its working properly, even if I add build_only: false to the testcase.yaml file here is the result,



Can I assume that this unit test worked properly? because I didn't added the unit_testing platform.
What is the difference between unit_testing platform and native platforms?
Can I use that unit_testing platform for my all types of unit testing, I'm also open for another recommendations?
Why its not execute in native platforms?

[dejans]

Hi,

You can start by looking at Unit Testing Quick Start. "type:unit" in test yaml means it is supposed to only run on "unit_testing" platform. This is a setup to load only minimal functionality from zephyr to allow building apps and executing them with mocks on your host. Please note that native_posix is depricated (native_sim replaces it). Native_sim is a "full" platform, and that is why a lot of stuff will be missing when one starts with "unit_testing". Unit tests (using the board unit_testing) are intended to run on your host system and thus are a bit different than running on embedded target. There are some tests in Zephyr that can be built both using unit_testing as the target board or targeting an embedded target. Here is one example of such tests. 

Best regards,
Dejan