Zigbee end device with low RSSI removed from the network

[Laura M]

Hello,

we implemented Zigbee coordinator. Few end devices left the network and we know that the issue is our side because the same devices work well with other coordinators.

This issue is rare and it happens only with devices that were reporting low RSSI values (we don't know how to replicate the issue and we have no logs).

This issue is similar to the one described here but there is no conclusion there.

We use nrf52840 with nrf5 sdk for thread and Zigbee v4.2.0 without soft device library.

We have the following questions:

1. Are you aware of any known issue related to described behaviour, with this SDK?
2. Recently we added the management of ZB_ZDO_SIGNAL_DEVICE_UPDATE in the case where status is ZB_DEVICE_LEFT (zb_zdo_signal_device_update_params_t), previously we just ignored this signal. Can this be the issue?
3. Can this signal be triggered even if there are no routers in the network?

Best regards.

Laura

[Sigurd Hellesvik]

Hi,

This issue is rare and it happens only with devices that were reporting low RSSI values (we don't know how to replicate the issue and we have no logs).

These make it hard to find the issue. You probably already know, but for the record:
It would be a lot easier to debug the issue if

• You were able to reproduce the issue
• If we got logs from the issue.
• If we got a sniffer trace from the issue (nRF Sniffer for 802.15.4)

Are you aware of any known issue related to described behaviour, with this SDK?

Nothing comes to mind

Recently we added the management of ZB_ZDO_SIGNAL_DEVICE_UPDATE in the case where status is ZB_DEVICE_LEFT (zb_zdo_signal_device_update_params_t), previously we just ignored this signal. Can this be the issue?

It should not be an issue, but I suggest that you try to disable it for a test and see if that fails still.

Can this signal be triggered even if there are no routers in the network?

It should still be as long as you got Coordinators.

Are there any Routers in your network?

Regards,
Sigurd Hellesvik

[Laura M]

Hello Sigurd,

there shouldn't be routers but we are not 100% sure for some setups.

Considering that we implemented Zigbee coordinator, when should we get ZB_ZDO_SIGNAL_DEVICE_UPDATE with status ZB_DEVICE_LEFT and when we should receive instead ZB_ZDO_SIGNAL_LEAVE_INDICATION with status ZB_NWK_LEAVE_TYPE_RESET ?

Regards.

Laura

[Sigurd Hellesvik]

Did some testing, and this is what we see:

If ZC is parent for a ZED that leaves:

• ZB_ZDO_SIGNAL_LEAVE_INDICATION is generated on ZC

If ZR is parent for a ZED that leaves:

• ZB_ZDO_SIGNAL_LEAVE_INDICATION is generated on ZR
• ZB_ZDO_SIGNAL_DEVICE_UPDATE is generated on ZE with status 2
• ZB_ZDO_SIGNAL_DEVICE_UPDATE is generated on ZC with status 2

It seems like ZB_ZDO_SIGNAL_LEAVE_INDICATION is always generated on the device that is the parent.
And that ZB_ZDO_SIGNAL_DEVICE_UPDATE is only genreated if a router was the parent.

Does this answer your question?

[anusha_14]

Hi Sigurd,

I would like to request you to reopen this ticket as we now have the Zigbee logs for the issue. For this particular ZED that left the network, Nordic Zigbee coordinator was seeing a low RSSI in the range -80 dBm to -95 dBm.  I have attached the log here and here are some details of the log: ZEDLeavesNetwork.pcapng

Network key:18:25:45:BB:F2:D0:58:C8:8D:0C:2F:7A:C2:A5:BF:C2 (Reverse Order)

TC Link key: 5A:69:67:42:65:65:41:6C:6C:69:61:6E:63:65:30:39 (Normal Order)

Packet 293056 - ZED sends rejoin request. 

Packet 293109 - ZED sends another rejoin request.

Packet 293116 - Coordinator sends rejoin response with status success.

Packet 293119 - Coordinator sends rejoin response with status success.

Packet 293130 - Coordinator sends rejoin response with status success.

Packet 293134 - ZED sends device announcement packet.

Packet 293136, 293138, 293139 - ZED sends End Device Timeout Request.

Packet 293141 - ZED sends Match Descriptor Request for OTA Upgrade cluster in Input Cluster List.

Packet 293143 - ZED sends device announcement packet.

Packet 293144 - ZED sends Match Descriptor Request for OTA Upgrade cluster in Input Cluster List.

Packet 293146 - ZED sends IAS Zone notification.

Packet 293148, 293149 - ZED sends data request.

Packet 293151 - Coordinator sends Leave request, with rejoin set to False.

Let me know if you need any other details to debug this further. Also, please let me know if you prefer to create a new ticket to track this.

Regards,

Anusha

[Elfving]

Hi Anusha,

It does seem a bit similar to KRKNWK-12017 (the link is for NCS, but Zigbee SDK v4.2.0 uses the same ZBOSS stack as the NCS versions listed there, v3.11). Could you try implementing the workaround listed there?

we implemented Zigbee coordinator.

So is the Coordinator the nRF running our Zigbee stack, and the ZEDs are not?

Regards,

Elfving

[Laura M]

Hello Elfving,

Elfving said:

So is the Coordinator the nRF running our Zigbee stack, and the ZEDs are not?

yes, correct. ZED doesn't implement Nordic Zigbee stack.

Elfving said:

It does seem a bit similar to KRKNWK-12017

It doesn't seem related in my opinion. The workaround you suggested is intended for a ZED and it activates if signal ZB_ZDO_SIGNAL_LEAVE is triggered. Here instead we implemented Zigbee coordinator (ZC) and signal ZB_ZDO_SIGNAL_LEAVE "is generated when the device itself has left the network by sending leave command" (as per Nordic documentation). So this signal doesn't seem related to our scenario.

Let me know if you don't agree with my previous statement or if you have any other hint.

Thank you.

Laura

[Elfving]

Laura M said:

It doesn't seem related in my opinion.

Yeah, as the nRF is the coordinator this doesn't seem related. I agree.

You have a sniffer trace of this, but would it also be possible to get the logs from the coordinator?

Regards,

Elfving

[anusha_14]

Hi Elfving,

Thanks for your response.

Unfortunately, we do not have logs from the coordinator as the Nordic FW was running without debug prints enabled. It will be very difficult to get logs since the issue happens quite randomly.

This issue was first reported from field and we did not have any zigbee logs. 

In order to get zigbee logs, we had to create multiple setups, with zigbee sniffer logging zigbee traffic and wait several weeks until the issue occurred in one of the setups. 

If there are specific questions related to our implementation of Zigbee coordinator, we can provide those details if it helps in debugging this issue.

Additionally, it would be good to understand some points from zigbee logs, which may give a clue regarding the cause of the issue?

1. We can see that the ZED sends Match Descriptor Request for OTA Upgrade cluster in Input Cluster List. Note that our Zigbee coordinator has not implemented this cluster. Could this be a reason for coordinator sending leave request?

2. Do you think implementing this OTA cluster in Nordic base descriptor could help in avoiding this issue?

3. Is there a way to disable zigbee stack from sending leave request on its own?

Regards,

Anusha

[Elfving]

I see that you've also opened another case for this. Is there a clear difference between the two issues, or should we just have one open to make it easier?

If this is in fact a new bug of some sort, it might be hard to get find the resources to address it now, as the nRF5 Zigbee SDK after all is deprecated. But we'll of course try to help out.

Regards,

Elfving

[anusha_14]

Hi Elfving,

Yes, I have opened another case where the end result is the same (i.e ZED leaving network). Even though the end result is the same, if you analyse the zigbee logs, there is a clear difference:

1. In this ticket https://devzone.nordicsemi.com/f/nordic-q-a/118485/zigbee-end-device-with-low-rssi-removed-from-the-network/530623, the issue occurs when the ZC sees low RSSI from the ZED (in the range of -80 dBm or lower). In this case, the ZED leaves the network because the Zigbee coordinator sent the leave request command to ZED.

2. In this ticket https://devzone.nordicsemi.com/f/nordic-q-a/119559/zigbee-end-device-sending-leave-command-after-receiving-transport-key-command, the issue occurred randomly and so far has occurred only once. The ZC was very close to ZED and it was seeing a good RSSI of ZED , maybe -50 dBm or higher. In this case, ZED sends the leave command and then leaves the network.

If it is hard to get resources, I would prefer if you can address this issue where ZED with low RSSI leaves network first as we have several instances of this issue happening in field. 

For the issue where ZED sends leave command, as an initial investigation, it would help in understanding with some level of certainty if the issue could be from ZC (that uses Nordic Zboss stack) or from ZED (does not use Zboss stack).

Regards,

Anusha

[anusha_14]

Hi Elfving,

I would like to check if you have an update on this ticket?

Regards,

Anusha