Matter Test Harness tests failing

[Tiit]

Hi,

I'm having trouble with the Matter Test Harness randomly failing tests and failing to commission my device. I have tried this with both my own Matter device and the Window Cover sample from the 3.0.1 SDK. I have the TH set up on a Pi 5 with the Nordic dongle as the RCP. I have turned off all other Thread devices in my office. I have tried running the code on the nRF52840-DK and my own hardware but things still randomly fail.

I'm attaching a very short test run I did against my own hardware and code but it only ran a few tests as I disabled all the rest. I can see that the device did get commissioned in this case (which it doesn't every time) and it did send matter packets to the device for each test. But it failed to "Run entire test" for some reason and I'm not sure where to go from here. The whole Matter certification thing seems extremely complex and difficult to get into. Is there somewhere I can learn more about this. As I understand I should use PICS tool to create all the test cases for my device (I did that), then set up a Matter TH and feed in the PICS files and then that should test my device. But when I do that most tests fail and some seem to ask me to do manual testing. The ones asking for manual tests seem broken as they tell me to subscribe to on-off clusters which my device doesn't even have. 

Please advise if this is even the right forum for this question. Or should I contact CSA or my test lab for this.

Tiit

device.logUI_Test_Run_2025_08_14_14_51_21.log

[Elfving]

Hi Tiit,

We do offer a lot of documentation on this,  which looks like a good idea to read up on. Let me know if this doesn't help answering your questions. I am not that knowledgeable about this myself, though I'd be happy to ask someone if run into any issues.

Regards,

Elfving

[Tiit]

Hi Elfving,

I did already read that and also much of the CSA documentation. But I'm still stuck with this. The error logs I attached show failures but I can't tell from the logs why the tests are failing. I think it may be because my PICS files and the things my device report do not match up for some reason. But I can't find a log or anything that would tell me where the issue is specifically. As far as I understand I have two separate tools where I need to define my device data model - PICS tool and ZAP tool. And I have to manually make sure they match exactly. Is this correct? Or is there a way I can do this in one tool and have the other import and use the same definition somehow? Also - can I somehow see the OTBR logs while the Test Harness is doing the automatic tests? And another question I have is how do I select which test I need to run in the Test Harness? Or more specifically - which tests will my ACL run on my device?

Tiit

[Elfving]

TC_IDM_10_2 seems to fail because FeatureMap attribute is a different data type than the TH expects and TC_IDM_10_4 fails because the wildcard read returns incomplete TLV.

I had a talk with someone who knows this a bit better than I do, who believes that both of these issues stem from the same problem. Maybe the PICS you provided to the TH are wrong/doesn't correspond with what you actually have on the device. A customer with a similar problem in the past had ZAP files that didn't match the PICS or that the ZAP files were generated with a version of the ZAP tool that used a different feature level than the SDK.

Here is a PICS tool but I believe you have to do things manually there as well, and there is a user guide for TH here.

Regards,

Elfving

[Tiit]

I have tried to match the ZAP and PICS the best I can. That process is incredibly not user friendly though. Why can't I just define the data model in one tool and have the other tool read/import that?

I think one of the reason for all these failures is that I can't seem to define the endpoint for each cluster anywhere. None of the guides show how to do this unfortunately. I have 3 endpoints defined in ZAP: endpoint 0 as the standard thing, endpoint 1 for the window cover and battery status, endpoint 2 for the environmental sensors. But the TH tried to run all sensor cluster tests on endpoint 1. I tried using the drop-down selector in PICS tool to define the endpoint, but that creates files that are identical to when I don't use that. I tried manually editing the test configuration JSON think in the TH Project Config but that errors out whenever I tough the endpoint configuration. How can I tell the Matter TH to run a test on a specific endpoint?

Tiit 

[hotbeefcurry]

Hey Tiit,

Within your project configuration on the test harness, you should have a test parameters section. You can define the endpoint there. Kinda like this.

{
  "config": {
    "test_parameters": {
      "endpoint": 1
    },
    "network": {
      "wifi": {
        "ssid": "testharness",
        "password": "wifi-password"
      },

[Elfving]

Yeah I agree that the process is cumbersome, though unfortunately this is currently the way it is.

Did you have a look at the suggestion from another user here above? I interpreted your lack of response to that as it working to some degree?

Regards,

Elfving