Communication lost with Zigbee

[mau]

Hi to everybody,

I'm developing over nRF5 SDK v. 3.2.0 a Zigbee node in router role. In my network I've four of my nodes plus several commercial led light driver and a coordinator.

After random amount of time (from one day, to one week) some of my nodes lost communication with coordinator and never come back to communicate, the only way to recover is reboot. (rejoin is not needed).

I attach the sniffer traces, so you can see what happens; unfortunately I can't caught the exact moment in which the issue start because the long time observation period.

trace1 trace2

You have to set this preconfigured key in Wireshark in order to view the trace: 48:35:70:4b:7c:60:63:f1:4f:89:66:f5:04:65:ed:7d

In order to understand the issue I provide to you my devices long and short addresses:

F4:CE:36:53:37:52:8C:10 CBB2 Currently working
F4:CE:36:31:19:B7:A3:25 5F7A Connection lost
F4:CE:36:84:13:85:21:71 BCBC Connection lost
F4:CE:36:8C:78:03:53:9D 30A9 Connection lost

As you can see there are several messages from my devices, so these nodes are not unjoined, but the coordinator ignore every message from them, even "Device Announcement" messages.
Affected nodes only send this kinds of messages: "Network address request", "Link status", "Device Announcement", "Route request".

I can't understand the status that these nodes assumes, it seems that they lose the routing table.

[Marjeris Romero]

Hi,

Sorry for the late reply. I took a look at the traces but without really understanding what could be the reason behind the nodes leaving the network. How do you know they have lost communication with the coordinator? Do they send a Leave message or do they get unresponsive somehow?

Anyways I have also forwarded this case to our Zigbee team to see if they may have any more insight about what could be happening.

Best regards,

Marjeris

[mau]

Hi Marjeris,

Marjeris Romero said:

How do you know they have lost communication with the coordinator?

The coordinator no longer receives any configured report or "device announce" message from affected nodes and when I try to read an attribute (I do it using CLI on coordinator) I see only a "route request" message from coordinator; there is no "route reply" from affected node and coordinator do not proceed to read the attribute.

Best regards,

Maurizio

[Marjeris Romero]

Hi Maurizio,

Just to update you here. I am waiting for some input from R&D for ideas on how to proceed here, but as I see it it looks like it would be difficult for us to recreate the issue when so many third party devices are involved.

I was thinking, what happens if the affected nodes are in proximity of the coordinator? Does the coordinator reply to the route request of the affected nodes then? The nodes need a route request reply in order to send reports, so that's why they stop reporting. Maybe you can sniff the traffic between the coordinator and one of the affected nodes when the node is placed in proximity to the coordinator?

BR,

Marjeris

[mau]

Hi Marjeris,

Marjeris Romero said:

Just to update you here. I am waiting for some input from R&D for ideas on how to proceed here, but as I see it it looks like it would be difficult for us to recreate the issue when so many third party devices are involved.

thank you.

Marjeris Romero said:

I was thinking, what happens if the affected nodes are in proximity of the coordinator? Does the coordinator reply to the route request of the affected nodes then?

affected nodes are already in direct communication with coordinator. Traces attached to the case was sniffed at approximately 2 meters from coordinator.

Best regards,

Maurizio

[Marjeris Romero]

Hi Maurizio,

mau said:

Traces attached to the case was sniffed at approximately 2 meters from coordinator.

OK, thanks for the clarification.

After consulting with R&D this is what they propose:

They would like you to experiment and try to figure out the smallest network size when this occurs. This is important to reduce the size of information we need to investigate the matter further.

While doing so you should:

• Instrument the network with a sniffer that perform traffic capture continuously.
• Take ZBOSS trace logs from nodes in question so that we can decode logs.

This page should cover on how to implement ZBOSS debug tracing: https://infocenter.nordicsemi.com/index.jsp?topic=%2Fsdk_tz_v4.1.0%2Fzigbee_prog_principles.html&cp=7_3_3_3&anchor=zigbee_debugging

I am sorry this has taken so long to figure out, but you see without more information we are a bit in the darkness here, specially when this is a larger network and the issue is not occuring with only ZBOSS devices.

Best regards,

Marjeris

[mau]

Hi Marjeris,

Marjeris Romero said:

They would like you to experiment and try to figure out the smallest network size when this occurs. This is important to reduce the size of information we need to investigate the matter further.

We are already working to do this

Marjeris Romero said:

Instrument the network with a sniffer that perform traffic capture continuously.

This is not a problem for the network we are making.

Marjeris Romero said:

Take ZBOSS trace logs from nodes in question so that we can decode logs.

This is a little bit difficult for me because I've no experience with logger module and also our test network is physically in another location managed to the customer.

BR,

Maurizio

[Marjeris Romero]

Hi Maurizio,
For enabling ZBOSS logging there are only a few changes you need to do in sdk_config.h:

• Set ZIGBEE_TRACE_LEVEL to 4 (debug)
• Set ZIGBEE_TRACE_MASK to the value which track all subsystems (TRACE_SUBSYSTEM_ZCL| TRACE_SUBSYSTEM_ZDO|TRACE_SUBSYSTEM_NWK) which will be 0x0058.

The log output would look quite cryptic, but can be decoded by our Zigbee developers.

If the network is in another location, the best thing would be if you can try to recreate the issue with a minimal setup at your end using a DK and collect the log output from the DK.

mau said:

Instrument the network with a sniffer that perform traffic capture continuously.

What I mean is that for debugging purpuses you should set a continuous traffic capture of the network together with the ZBOSS trace logs. Not to implement the sniffer as a part of your network.

Best regards,

Marjeris