Matter Test Harness Python tests failing commissioning

[Tiit]

Hi,

I have now run most of the YAML test successfully on the Matter TH and while I usually need to restart the TH between runs it seems fairly reliable now. Before sending to certification labs I also wanted to run the IDM-10.4 (and other IDM tests) to be sure that my PICS files are correct. But whatever I do I can not get the TH to pass any of the Python tests. It always fails to commission the DUT. I can see that BLE packets are exchanged and communications are happening but it never succeeds. 

What logs can I send you to find out what is going wrong here? There are now so many different logs that I won't even try to send anything until you guide me in the right direction. I tried looking at similar open tickets here and it seems I'm not alone with this issue. But none of them really give me an idea what I should do to solve this. I'm trying to run either just the TC-IDM-10.4 or a small list of the 10.x tests but they all always fail.

idm_2025_09_25_08_51_34.log

I have attached a log from a recent test run here but I can't see what is going wrong from that.

Tiit

[Amanda Hsieh]

Hi, 

What NCS version are you using?

It's worth checking out this post. 

Regards,
Amanda H.

[Tiit]

Hi,

I'm using 3.0.1. And TH Version: v2.12+spring2025.

I did see that post and according to https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/protocols/matter/index.html I should have the right version of everything. Withe 3.0.1 I should be using Matter 1.4.1 and that is specified to work with TH v2.12+spring2025.

Tiit

[Amanda Hsieh]

Hi, 

Try to perform a factory reset on the device between each test case and complete commissioning (based on Test Harness prompts), or keep the device commissioned and select the correct answers to the prompts at the beginning of each test case.

-Amanda H.

[Tiit]

Hi,

I already tried all that. The TH is so bad at running multiple tests in a row that I always just select one and factory reset my device and also power cycle the whole TH just to get some sort of stability even with the YAML tests.

But with the Python tests I have not been able to successfully commission the device even once. I'm not sure why that would be as nothing else changed. I'm using the exact same code on my device. I'm using the exact same TH. Same positioning of the devices. And I can reliably get YAML tests to work and just as reliably Python to fail. chip-tool works nicely and I can read and write all the clusters that I have tried.

Tiit

[Amanda Hsieh]

Please share the logs from running the test one by one, since the provided logs look like you tried to commission the device, which is already commissioned.

Also, don't know what the discriminator and setupin code of the device, suggest to test_paramters (in Test Harness project) adds

"payload": "<qr code payload e.g.MT:QJL14UDB00JO.86CO00>",
 "bool-arg": "use_pase_only:false"

[Tiit]

Hi Amanda,

I have the DUT setup parameters defined like this right now:

"dut_config": {
      "discriminator": "1",
      "setup_code": "20250926",
      "pairing_mode": "ble-thread",
      "chip_timeout": null,
      "chip_use_paa_certs": false,
      "trace_log": true,
      "enhanced_setup_flow": null
    }

Where in this structure should I add the setup payload part?

I eventually got the Python test to run. I was working on another issue (ZAP tool forces some attributes to use External storage and that causes reading the Feature Map attribute to error out) and fixing that seems to have fixed this here too for now. I don't have time to test this theory but my gut says that if a Feature Map read fails the whole test will fail with no real description of what went wrong. 

We can consider this closed as I'm moving on to other issues now.

Tiit