Matter Test Harness tests failing

Hi Tiit,

We do offer a lot of documentation on this,  which looks like a good idea to read up on. Let me know if this doesn't help answering your questions. I am not that knowledgeable about this myself, though I'd be happy to ask someone if run into any issues.

Regards,

Elfving

Hi Elfving,

I did already read that and also much of the CSA documentation. But I'm still stuck with this. The error logs I attached show failures but I can't tell from the logs why the tests are failing. I think it may be because my PICS files and the things my device report do not match up for some reason. But I can't find a log or anything that would tell me where the issue is specifically. As far as I understand I have two separate tools where I need to define my device data model - PICS tool and ZAP tool. And I have to manually make sure they match exactly. Is this correct? Or is there a way I can do this in one tool and have the other import and use the same definition somehow? Also - can I somehow see the OTBR logs while the Test Harness is doing the automatic tests? And another question I have is how do I select which test I need to run in the Test Harness? Or more specifically - which tests will my ACL run on my device?

Tiit

TC_IDM_10_2 seems to fail because FeatureMap attribute is a different data type than the TH expects and TC_IDM_10_4 fails because the wildcard read returns incomplete TLV.

I had a talk with someone who knows this a bit better than I do, who believes that both of these issues stem from the same problem. Maybe the PICS you provided to the TH are wrong/doesn't correspond with what you actually have on the device. A customer with a similar problem in the past had ZAP files that didn't match the PICS or that the ZAP files were generated with a version of the ZAP tool that used a different feature level than the SDK.

Here is a PICS tool but I believe you have to do things manually there as well, and there is a user guide for TH here.

Regards,

Elfving

I have tried to match the ZAP and PICS the best I can. That process is incredibly not user friendly though. Why can't I just define the data model in one tool and have the other tool read/import that?

I think one of the reason for all these failures is that I can't seem to define the endpoint for each cluster anywhere. None of the guides show how to do this unfortunately. I have 3 endpoints defined in ZAP: endpoint 0 as the standard thing, endpoint 1 for the window cover and battery status, endpoint 2 for the environmental sensors. But the TH tried to run all sensor cluster tests on endpoint 1. I tried using the drop-down selector in PICS tool to define the endpoint, but that creates files that are identical to when I don't use that. I tried manually editing the test configuration JSON think in the TH Project Config but that errors out whenever I tough the endpoint configuration. How can I tell the Matter TH to run a test on a specific endpoint?

Tiit 

Hey Tiit,

Within your project configuration on the test harness, you should have a test parameters section. You can define the endpoint there. Kinda like this.

{
  "config": {
    "test_parameters": {
      "endpoint": 1
    },
    "network": {
      "wifi": {
        "ssid": "testharness",
        "password": "wifi-password"
      },