zigbee: similar issue as KRKNWK-12017

RE: zigbee: similar issue as KRKNWK-12017

Rogger — Thu, 15 Aug 2024 18:57:36 GMT

Have you been able to solve the issue with a device reset after parent link failure?

RE: zigbee: similar issue as KRKNWK-12017

Cheb — Tue, 13 Feb 2024 07:56:02 GMT

[quote userid="92402" url="~/f/nordic-q-a/107861/zigbee-similar-issue-as-krknwk-12017/468380"]In that case, you can reset the device on the first ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE. Please be aware that this will make the device reset in situations where it might have been able to find the parent again or rejoin under a new parent, but it should be able to rejoin successfully on restart.[/quote]

Thanks we have this logic under test.

RE: zigbee: similar issue as KRKNWK-12017

Marte Myrvold — Mon, 12 Feb 2024 10:55:22 GMT

Hi Sebastien,

In that case, you can reset the device on the first ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE. Please be aware that this will make the device reset in situations where it might have been able to find the parent again or rejoin under a new parent, but it should be able to rejoin successfully on restart.

Best regards,
Marte

RE: zigbee: similar issue as KRKNWK-12017

Cheb — Mon, 05 Feb 2024 16:22:27 GMT

[quote userid="92402" url="~/f/nordic-q-a/107861/zigbee-similar-issue-as-krknwk-12017/467593"]Do you only get ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE once or do you get this signal multiple times when it fails? The best option is to have a failure counter that checks how many times the device has received ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE, and then have a threshold for when it should reset. [/quote]

When it fails ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE happens once and there are no more signals received from the zboss signal handler.

Regards,

Sebastien

RE: zigbee: similar issue as KRKNWK-12017

Marte Myrvold — Mon, 05 Feb 2024 14:13:11 GMT

Hi,

[quote user="Cheb"]Can you check with zigbee team if it would be safe to reset the device for any parent link failure ?
KRKNWK-12017 doesn't seem top be enough.[/quote]

Do you only get ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE once or do you get this signal multiple times when it fails? The best option is to have a failure counter that checks how many times the device has received ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE, and then have a threshold for when it should reset.

Best regards,
Marte

RE: zigbee: similar issue as KRKNWK-12017

Cheb — Thu, 01 Feb 2024 13:14:42 GMT

Hi,

Some feedback from the tests.

We added some counters that we can get via BLE and this what we observed.
One device ran a few times into the forced rejoined and finally hit the broken rejoin case and rebooted (good)
One other device ran into the forced rejoin once and that's all. Device not visible on the network anymore.
It seems to me that even if not in the broken rejoin logic the ZBOSS stack can be KO.

Can you check with zigbee team if it would be safe to reset the device for any parent link failure ?
KRKNWK-12017 doesn't seem top be enough.

FYI, a few weeks ago we managed to reproduce with the serial output and when the parent link failure happened there were no ZB sleeping event at all anymore which makes me think the stack was dead.

Thx

RE: zigbee: similar issue as KRKNWK-12017

Marte Myrvold — Thu, 01 Feb 2024 10:34:36 GMT

Hi,

[quote user="Cheb"]KRKNWK-12017 is clearly a production stop issue. Why hasn't it be fixed by ZBoss stack ? Why the workaround isn't officially integrated ?[/quote]

The issue has been reported to DSR. However, the nRF5 SDK for Thread and Zigbee is deprecated and will not be upgraded.

Best regards,
Marte

RE: zigbee: similar issue as KRKNWK-12017

Cheb — Wed, 31 Jan 2024 08:11:25 GMT

FYI
In the current product we have 2 watchdogs, 1 periodic reset every 12 hours and the KRKNWK-12017 workaround to manage having a stable product

RE: zigbee: similar issue as KRKNWK-12017

Cheb — Wed, 31 Jan 2024 08:09:25 GMT

[quote userid="92402" url="~/f/nordic-q-a/107861/zigbee-similar-issue-as-krknwk-12017/466874"]Yes, that should take care of both types of failures. However, I recommend testing it to verify that it works as expected.[/quote]

Let me be clear here. It's the second major issue we had with nrf52840 and Zigbee SDK 4.2.0 (released recently which was supposed to be a stable production release)

We had to recall products because of stability issues of the SDK / ZBOSS stack.
With the closed source ZBOSS stack and the binary trace it's impossible to debug by ourselves and it's time and resource consuming to debug with the support.

KRKNWK-12017 is clearly a production stop issue. Why hasn't it be fixed by ZBoss stack ? Why the workaround isn't officially integrated ?

I really like the software ecosystem, the SDK and such but quality should be highly prioritized in Nordic.

So in conclusion, yes, don't worry, we will test.

RE: zigbee: similar issue as KRKNWK-12017

Marte Myrvold — Wed, 31 Jan 2024 07:46:52 GMT

Hi,

Yes, that should take care of both types of failures. However, I recommend testing it to verify that it works as expected.

Best regards,
Marte

RE: zigbee: similar issue as KRKNWK-12017

Cheb — Tue, 30 Jan 2024 15:38:27 GMT

So to handle both cases you would recommend this ?

case ZB_NLME_STATUS_INDICATION: {
    zb_zdo_signal_nlme_status_indication_params_t *nlme_status_ind =
        ZB_ZDO_SIGNAL_GET_PARAMS(sig_hndler, zb_zdo_signal_nlme_status_indication_params_t);
    if (nlme_status_ind->nlme_status.status == ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE) {

        /* Check for broken rejoin procedure and restart the device to recover. */
        if (stack_initialised && !joining_signal_received) {
            zb_reset(0);
        } else {
            bdb_start_top_level_commissioning(ZB_BDB_NETWORK_STEERING);
        }
    }
    break;

RE: zigbee: similar issue as KRKNWK-12017

Marte Myrvold — Tue, 30 Jan 2024 14:48:02 GMT

Hi,

We have a workaround for KRKNWK-12017:

Complete the following steps to detect when the rejoin procedure breaks and reset the device:

Introduce helper variable joining_signal_received.
Extend zigbee_default_signal_handler() by completing the following steps:
1. Set joining_signal_received to true in the following signals: ZB_BDB_SIGNAL_DEVICE_FIRST_START, ZB_BDB_SIGNAL_DEVICE_REBOOT, ZB_BDB_SIGNAL_STEERING.
2. If leave_type is set to ZB_NWK_LEAVE_TYPE_REJOIN, set joining_signal_received to false in the ZB_ZDO_SIGNAL_LEAVE signal.
3. Handle the ZB_NLME_STATUS_INDICATION signal to detect when End Device failed to transmit packet to its parent, reported by signal’s status ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE.

See the following code snippet for an example:

/* Add helper variable that will be used for detecting broken rejoin procedure. */
/* Flag indicating if joining signal has been received since restart or leave with rejoin. */
bool joining_signal_received = false;
/* Extend the zigbee_default_signal_handler() function. */
case ZB_BDB_SIGNAL_DEVICE_FIRST_START:
    ...
    joining_signal_received = true;
    break;
case ZB_BDB_SIGNAL_DEVICE_REBOOT:
    ...
    joining_signal_received = true;
    break;
case ZB_BDB_SIGNAL_STEERING:
    ...
    joining_signal_received = true;
    break;
case ZB_ZDO_SIGNAL_LEAVE:
    if (status == RET_OK) {
        zb_zdo_signal_leave_params_t *leave_params = ZB_ZDO_SIGNAL_GET_PARAMS(sig_hndler, zb_zdo_signal_leave_params_t);
        LOG_INF("Network left (leave type: %d)", leave_params->leave_type);

        /* Set joining_signal_received to false so broken rejoin procedure can be detected correctly. */
        if (leave_params->leave_type == ZB_NWK_LEAVE_TYPE_REJOIN) {
            joining_signal_received = false;
        }
    ...
    break;
case ZB_NLME_STATUS_INDICATION: {
    zb_zdo_signal_nlme_status_indication_params_t *nlme_status_ind =
        ZB_ZDO_SIGNAL_GET_PARAMS(sig_hndler, zb_zdo_signal_nlme_status_indication_params_t);
    if (nlme_status_ind->nlme_status.status == ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE) {

        /* Check for broken rejoin procedure and restart the device to recover. */
        if (stack_initialised && !joining_signal_received) {
            zb_reset(0);
        }
    }
    break;
}

Please note that this code snippet is from the nRF Connect SDK, but it should be very similar in nRF5 SDK for Thread and Zigbee.

Best regards,
Marte

RE: zigbee: similar issue as KRKNWK-12017

Cheb — Tue, 30 Jan 2024 14:16:16 GMT

Thanks for your input.

How to be sure we're not running into KRKNWK-12017 in which the ZBOSS stack requires a reset ?

RE: zigbee: similar issue as KRKNWK-12017

Marte Myrvold — Tue, 30 Jan 2024 13:12:09 GMT

Hi Sebastien,

[quote user=""]Could you recommend a procedure to fix this ? Maybe trigger a rejoin when the ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE occurs ?[/quote]

Yes, since you are able to trace it to a stack signal, the best workaround would be to trigger a rejoin when the signal occurs. You can add a case for ZB_NLME_STATUS_INDICATION in zboss_signal_handler that checks if the status is ZB_NWK_COMMAND_STATUS_PARENT_LINK_FAILURE. To rejoin, you can call bdb_start_top_level_commissioning(ZB_BDB_NETWORK_STEERING).

If you are able to reproduce the issue and collect sniffer logs, please let me know.

Best regards,
Marte