OTA Upgrade Issue (Matter1.1->Matter1.3) Causes Invalid CASE Parameter Error in Chip-Tool Commands

[SemiBrain]

After the Matter device is upgraded from version 1.1 to 1.3 via OTA, the Thread connection is functioning normally., but it fails to process chip-tool commands, and chip-tool returns an "Invalid CASE parameter" error. 
Is this an issue related to Matter version compatibility? How can it be resolved?

Following is the logs:

Device Error Log

[00:01:01.249,847] <inf> chip: [EM]>>> [E:53123r S:0 M:182431476] (U) Msg RX from 0:312C04970EEA4A9E [0000] --- Type 0000:30 (SecureChannel:CASE_Sigma1)
[00:01:01.250,274] <inf> chip: [IN]CASE Server received Sigma1 message . Starting handshake. EC 0x200058b0
[00:01:01.250,823] <inf> chip: [EM]<<< [E:53123r S:0 M:118580912 (Ack:182431476)] (U) Msg TX to 0:0000000000000000 [0000] [UDP:[fd93:2a45:8b9a:1:8e51:d5ce:439:712a]:55401] --- Type 0000:10 (SecureChannel:StandaloneAck)
[00:01:01.252,258] <inf> chip: [SC]Received Sigma1 msg
[00:01:01.259,643] <inf> chip: [SC]CASE matched destination ID: fabricIndex 1, NodeID 0x0000000000000001
[00:01:01.312,774] <inf> chip: [EM]<<< [E:53123r S:0 M:118580913 (Ack:182431476)] (U) Msg TX to 0:0000000000000000 [0000] [UDP:[fd93:2a45:8b9a:1:8e51:d5ce:439:712a]:55401] --- Type 0000:40 (SecureChannel:StatusReport)
[00:01:01.314,453] <err> chip: [IN]CASE Session establishment failed: 71
[00:01:01.314,971] <err> chip: [EM]OnMessageReceived failed, err = 71
[00:01:01.452,331] <inf> chip: [EM]>>> [E:53123r S:0 M:182431477 (Ack:118580913)] (U) Msg RX from 0:312C04970EEA4A9E [0000] --- Type 0000:10 (SecureChannel:StandaloneAck)

chip-tool Error Log

[174056003.674] [36236:36238] [E] CHIP MessageCounter:203356144 not all Retransmittable on Exchange 62100i
[174056003.674] [36236:36238] [SC] Received error (protocol code 2) during pairing process: src/protocols/secure_channel/CASESession.cpp:2141: CHIP Error 0x00000054: Invalid CASE parameter
[174056003.674] [36236:36238] [IN] SecureSession[0xffff8400bc40]: Released - Type:2 LSID:48319
[174056003.674] [36236:36238] [-] src/protocols/secure_channel/CASESession.cpp:2141: CHIP Error 0x00000054: Invalid CASE parameter at ../../examples/chip-tool/commands/clusters/ModelCommand.cpp:67
[174056003.674] [36236:36238] [EM] <<< [E:62100i S:0 M:10693822 (Ack:203356144)] (U) Msg TX from 52DCD626597D166A to 0:0000000000000000 [0000] [UDP:[fd13:6e9a:686d:1:7474:2258:ff13:823f]:5540] --- Type 0000:10 (SecureChannel:StandaloneAck) (B:26)
[174056003.674] [36236:36238] [EM] Flushed pending ack for MessageCounter:203356144 on exchange 62100i
[174056003.675] [36236:36236] [CTL] Shutting down the commissioner
[174056003.675] [36236:36236] [CTL] Shutting down the controller
[174056003.675] [36236:36236] [IN] Expiring all sessions for fabric 0x1!!
[174056003.675] [36236:36236] [FP] Forgetting fabric 0x1
[174056003.675] [36236:36236] [TS] Pending Last Known Good Time: 2023-10-14T01:16:48
[174056003.675] [36236:36236] [TS] Previous Last Known Good Time: 2023-10-14T01:16:48
[174056003.675] [36236:36236] [TS] Reverted Last Known Good Time to previous value
[174056003.676] [36236:36236] [CTL] Shutting down the commissioner
[174056003.676] [36236:36236] [CTL] Shutting down the controller
[174056003.676] [36236:36236] [CTL] Shutting down the System State, this will teardown the CHIP Stack
[174056003.676] [36236:36236] [DMG] All ReadHandler-s are clean, clear GlobalDirtySet
[174056003.677] [36236:36236] [FP] Shutting down FabricTable
[174056003.677] [36236:36236] [TS] Pending Last Known Good Time: 2023-10-14T01:16:48
[174056003.677] [36236:36236] [TS] Previous Last Known Good Time: 2023-10-14T01:16:48
[174056003.677] [36236:36236] [TS] Reverted Last Known Good Time to previous value
[174056003.693] [36236:36236] [DL] Wrote settings to /tmp/chip_counters.ini
[174056003.693] [36236:36236] [DL] NVS set: chip-counters/total-operational-hours = 0 (0x0)
[174056003.693] [36236:36236] [DL] Inet Layer shutdown
[174056003.693] [36236:36236] [DL] BLE Layer shutdown
[174056003.693] [36236:36236] [DL] System Layer shutdown
[174056003.693] [36236:36236] [TOO] Run command failure: src/protocols/secure_channel/CASESession.cpp:2141: CHIP Error 0x00000054: Invalid CASE parameter

Regards,
Charles

[Amanda Hsieh]

Hi Charles, 

There was a change regarding the crypto backend between Matter 1.1 and Matter 1.3, or more precisely, from NCS 2.6.0. We moved from the mbedtls API to the PSA Crypto API. All Openthread keys are migrated automatically, without the need for any additional kconfig, but in the case of Matter, we couldn't do that, and an additional kconfig for such a migration is required; see the importantNote here: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/protocols/matter/index.html. 

Here is the migration guide for it: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/releases_and_maturity/migration/migration_guide_2.6.html#d1967e677, You can find there what it needs to use CONFIG_NCS_SAMPLE_MATTER_OPERATIONAL_KEYS_MIGRATION_TO_ITS=y to enable auto migration of keys from mbedtls to PSA API in Matter. There were more changes between those NCS versions, so I recommend reading all migration guides for NCS 2.6, 2.7, and 2.8.

Regards,
Amanda H.

[SemiBrain]

Hi Amanda,

Thank you for your reply.
After reviewing the documentation you provided, we noticed that when using the CONFIG_NCS_SAMPLE_MATTER_OPERATIONAL_KEYS_MIGRATION_TO_ITS option for key migration, there is a potential for migration failure. In such cases, the device will be reset by default.
We would like to better understand the likelihood of key migration failure and the possible reasons behind it. Would you be able to provide this data? Thanks!

Regards,
Charles

[Amanda Hsieh]

Hi, 

The migration process works in this way:

1. The algorithm seeks the fabric list to get all the current Matter fabrics the device is connected to.
2. tries to get an mbedtls key that is assigned to each fabric. The key is stored in the Zephyr's settings.
3. If the mbedtls key is found, then the algorithm copies the key to the PSA ITS.
4. Removes obsolete mbedtls key.

Note that the migration process will always be checking at the start of the application if there is a key to be migrated as long as the CONFIG_NCS_SAMPLE_MATTER_OPERATIONAL_KEYS_MIGRATION_TO_ITS is set. So it is recommended to perform migration, and then in the next firmware upgrade, disable it because all keys should be migrated. We have automated tests for this process, and we didn't see any problems.

One of our customers noticed an issue not with the migration process itself but the behaviour after following reboots. The device performs migration, and after a random reboot, it may perform a factory reset due to the failure. We haven't found a root case for it, and it is very hard to reproduce because it doesn't always occur, but for example, in 1 reboot per 1000 attempts. Due to that, to avoid such problems, we recommend disabling this config as quickly as possible after the migration.

Migration is important because without using the PSA ITS API, our customers cannot get the Thread Certificate by inheritance since NCS 2.6.0.

-Amanda H.

[SemiBrain]

Thank you for your detailed explanation.
Everything is much clearer now, and this has been incredibly helpful in resolving our issue. Much appreciated!

Regards,
Charles