Best tool for automated testing the firmware.

[nRF5xFanboy]

Hello Reader,

I wish to automate running tests on my new developed firmware, build daily in ci/cd (Azure), flashed into the embedded device (device under test).

What are candidates as tool /framework for this? 
I notice that zTest and twister is not popular for unknown reasons (any explanation why is also welcome.)

I currently using the toolchain of Nordic NCS 2.0x with MS VSCode.
Ideally the unit test(s) should be able to run manually (on demand) from within VScode.

Any tip/hint/idea's are most welcome!

Best regards,
      nRF5xFanBoy

[Elfving]

Hello again,

I can't get too much into our own workflow.

zTest is from zephyr upstream so it wouldn't be a part of NCS yet. I don't know why one would think that Twister isn't popular. - Twister is actually what I would recommend. There is a bit more info on unit testing and twister here (Though the zephyr documentation is better )

Regards,

Elfving

[nRF5xFanboy]

Thanks Elfving,

According to our junior developers the twister documentation is unclear and the answer they get on discord doesn't help  them really.

I am familiar with other test frameworks like google test framework
So I can't really help them with there questions.

So dear Elfving can you help out and tell a bit more ..

I will just place here a few questions from the junior developers :
- where does the testmain run (on target or on the PC) ?
- do you add CONFIG_ZTEST_NEW_API=y. to the application proj.conf?
- What does developers at Nordic use for unit testing?
- Where do you create testcases (inside the application folder?) 
- Where do you create testcases (inside the our custom zephyr extended library?) 
   because  the samples suggest it is a stand alone project (no functional unit to test comes with it? - that is confusing)    
- If ztest runs on target  how does it interact with the application (the test subject?)
- Does the testcase.yml exists next to the appplication.yml?
- We do not understand the interaction of the ztest code in the source code file itself?
- The zephyr testing framework is so unclear - are there good video's for beginners?
- if i place a testcase.yml next in the same folder as my_appl.yml will that work?
- Do you have a very easy sample that contain both a simple functionality (like blinky) that is being tested with a ztest .
    because zephyr documentation & samples either show a stand alone functionaliy or a sample testcase
     but never both - that is very confusing for most starters.
-  Do you have a proper unit testing examples?
-  Do you have a proper module testing examples?
-  Do you have a proper integration testing examples?
-  Do you have a proper stress test testing examples?
-  Do you have a proper test suite examples?
- Why does each sample not contains testcases? None of the sample.yml inclde 'test'.

Thanks Elfving

Kind regards, 
   nRF5xFanBoy

[Elfving]

nRF5xFanboy said:

- do you add CONFIG_ZTEST_NEW_API=y. to the application proj.conf?

I believe you would put it in a configuration file if ztest was what you were planning on using. Though Ztest is from zephyr upstream.

nRF5xFanboy said:

- What does developers at Nordic use for unit testing?

What specific tools we are using is something we can not disclose unfortunately.

nRF5xFanboy said:

- Where do you create testcases (inside the application folder?) 

In general we have very little available when it comes to samples and tutorials etc., unfortunately.

And one reason for this is that this is more related to Zephyr than NCS. Could you try to ask the rest of your Twister questions either on the Zephyr Project Discord  #twister channel? And you can raise issues here· zephyrproject-rtos/zephyr (github.com).

Regards,

Elfving