Assistance Required: Multiple Matter Certification Failures on nRF54L15 (NCS v3.1.1 / Matter 1.4.2)

[xiongwei.wang]

Dear Nordic Support Team,

I am currently performing Matter certification testing on our device based on the nRF54L15 using NCS v3.1.1 (Matter SDK 1.4.2). I am using the Matter Test Harness (v2.14+fall2025).

I have encountered several persistent failures during the "Mandatories" test suite run. Despite verifying our PICS and configuration, these tests continue to fail.

Below is a summary of the key failures and the specific error messages observed in the logs:

1. Device Type & Data Model Validation (TC-IDM-10.5)

Error: Invalid device type ID (out of valid range) But I just have a custom device type in endpoint 5 and I am testing the endpoint 0 now.

2. Access Control & Privileges (TC-ACE-2.3)
3. other failures:

TC-CADMIN-1.22 / 1.25.
TC-CCTRL-2.2 / 2.3.
TC-CGEN-2.2.
TC-CNET-4.10 / 4.12 / 4.16
TC-DA-1.2

How to analyze and solve them? Should I upload more information like PICS files ?

Environment Details:

Chip: nRF54L15
SDK: NCS v3.1.1
Matter Version: 1.4.2
Test Harness: v2.14+fall2025
Transport: BLE + Thread

Here is the total logs.

UI_Test_Run_2025_11_21_10_53_00.log

Best regards,

Xiongwei.Wang

[Elfving]

Hi, 

First of all, I should mention that we do offer a lot of documentation about this. And that the general way to fix these issues is to simply have a second look at your PICS and ZAP files. There is typically something missing, or maybe the ZAP files were generated with the wrong version of the tool, or rather a version that doesn't match the SDK.

Have you also seen this?

Get back to me if none of this helped, and I'll take another look.

Regards,

Elfving

[xiongwei.wang]

Hi,   Elfving

Sorry about it, I incorrectly assumed the latest version would be backward compatible. I will test it again. Thanks you very much.

Best regards,

Xiongwei.Wang 

[Elfving]

Glad to hear it! And no problem at all, feel free to open new tickets, no matter the problem

Regards,

Elfving

[xiongwei.wang]

Hi, Elfving

I am using the environment of v2.13+summer2025, but the test results still show the same issue.

Best regards,

Xiongwei.Wang

[Elfving]

I see. So you've checked your ZAP and PICS files?

Did you get your ZAP tool from NCS?

Regards,

Elfving

[xiongwei.wang]

Hi, Elfving,

Of course, the first time I used the west zap-gui of ncs 3.1.1, I downloaded the corresponding version of zap and have never manually downloaded it. I already have some idea about part of the issue and have made fixes.

Then, I submitted a new ticket regarding TH testing (with the same title as this ticket) about some issues that are not described in the following format:
Problem: ProblemSeverity.ERROR
test_name: xxxx
location:
Endpoint: 0,
Cluster: xxx
Command: xxx
problem: Problem reason xxxxx
spec_location:

These issues appear in a format like this:
Test results: Error 1, Executed 1, Failed 0, Passed 0, Requested 1, Skipped 0
There are many such entries in the file above, and I don’t know how they occurred. I’m not sure how to analyze them.

Then there is another issue here that I haven't found a solution for.

TH_second_controller checks the permission requirements for that command.
If the permission required for the command is HIGHER than the permission level being tested,
TH_second_controller sends the command to the DUT using default values.
Regardless of the command contents, the DUT should return an access error since access must be checked
before the command is processed. Receipt of an UNSUPPORTED_COMMAND error is a conformance failure.	Expected outcome: DUT returns UNSUPPORTED_ACCESS error
                 *
                 * Endpoint: 2
                 *
                 *******************************************************************

[MatterTest] 11-24 07:41:36.143 INFO 		"msgCounter" : 256304126,
[MatterTest] 11-24 07:41:36.144 INFO [Test] test_TC_ACE_2_3 FAIL
[MatterTest] 11-24 07:41:36.147 INFO 		"securityFlags" : 0,
[MatterTest] 11-24 07:41:36.153 INFO 		"sessionId" : 10278
[MatterTest] 11-24 07:41:36.154 INFO 	},
[MatterTest] 11-24 07:41:36.158 INFO 	"payload" : 
[MatterTest] 11-24 07:41:36.160 INFO ###########################################################
[MatterTest] 11-24 07:41:36.161 INFO 	{
[MatterTest] 11-24 07:41:36.161 INFO 		"decoded" : 
[MatterTest] 11-24 07:41:36.161 INFO Problems found:
[MatterTest] 11-24 07:41:36.162 INFO 		{
[MatterTest] 11-24 07:41:36.163 INFO ===============
[MatterTest] 11-24 07:41:36.164 INFO 			"mrp_ack" : ""
[MatterTest] 11-24 07:41:36.164 INFO 
Problem: ProblemSeverity.ERROR
    test_name: Command test - privilege 3
    location: 
       Endpoint: 0,
       Cluster:  42 (0x2a) OtaSoftwareUpdateRequestor
       Command:  0 (0x00)
    problem: Unexpected success sending command <class 'matter.clusters.Objects.OtaSoftwareUpdateRequestor.Commands.AnnounceOTAProvider'> with privilege 3
    spec_location: 

[MatterTest] 11-24 07:41:36.165 INFO 		},
[MatterTest] 11-24 07:41:36.166 INFO 
Problem: ProblemSeverity.ERROR
    test_name: Command test - privilege 4
    location: 
       Endpoint: 0,
       Cluster:  42 (0x2a) OtaSoftwareUpdateRequestor
       Command:  0 (0x00)
    problem: Unexpected success sending command <class 'matter.clusters.Objects.OtaSoftwareUpdateRequestor.Commands.AnnounceOTAProvider'> with privilege 4
    spec_location: 

 

New ticket:
Assistance Required: Multiple Matter Certification Failures on nRF54L15 (NCS v3.1.1 / Matter 1.4.2) - Nordic Q&A - Nordic DevZone - Nordic DevZone

Best regards,
Xiongwei.Wang

[xiongwei.wang]

Hi, Elfving,

When I execute the west zap-gui command, the output displays

Found ZAP 2025.5.14 (up to date) 

Does this mean the version is correct?

Best regards,
Xiongwei.Wang

[Elfving]

Hi again,

I see your other case is rather similar to this one, so let's continue this problem in the other one. 

Regards,

Elfving

[xiongwei.wang]

Hi,

No problem. Thanks a lot.

Best regards,
Xiongwei.Wang