ZigBee radio channel change

RE: ZigBee radio channel change

Marte Myrvold — Wed, 10 Jan 2024 12:42:05 GMT

Hi Laura,

I would not recommend changing the channel mask after the Zigbee stack has started. This might cause conflicting channels with the 802.15.4 radio since changing the Zigbee channel mask after the stack has started does not affect the 802.15.4 channel. So, for example, you might end up with a Zigbee channel mask that does not include the actual channel the 802.15.4 radio is operating on.

However, if you are careful and know what you are doing it should be possible. In addition to zb_set_channel_mask() you should also call zb_set_bdb_primary_channel_set(). And make sure that the current 802.15.4 channel is part of the channel mask. You can get the 802.15.4 channel with nrf_802154_channel_get().

Best regards,
Marte

RE: ZigBee radio channel change

Laura M — Tue, 09 Jan 2024 14:35:16 GMT

Hello Marte,

I have one last question about channel change, is it fine to use zb_set_channel_mask to change channel mask at runtime?

The idea is to:

set a channel mask that allows current channel and new channel
move to new channel as per our previous discussion
set a channel mask that allows only the new channel

Do you see any drawback? Except what we already wrote about devices not receiving the command to change channel because powered off for instance.

Best,

Laura

RE: ZigBee radio channel change

Marte Myrvold — Tue, 02 Jan 2024 10:19:51 GMT

Hi Laura,

It could be something else making the device initiate channel change, but it is difficult to say what without a sniffer log.

Best regards,
Marte

RE: ZigBee radio channel change

Laura M — Fri, 22 Dec 2023 14:30:10 GMT

Hello Marte,

I'm not sure if it is working because I left my setup running during the night and I saw that Nordic changed channel and that routers remained in previous channel. I don't have logs. I'll keep this monitored.

In the meanwhile, do you have any explanations for this? Devices in my setup are the same one listed above.

Thank you

Laura

RE: ZigBee radio channel change

Marte Myrvold — Thu, 21 Dec 2023 09:54:26 GMT

Hi Laura,

Please let me know if you still have issues with the channel change after disabling Mgmt_NWK_Update_notify 🙂

Best regards,
Marte

RE: ZigBee radio channel change

Laura M — Thu, 21 Dec 2023 09:37:19 GMT

Thank you Marte, this should really help

RE: ZigBee radio channel change

Marte Myrvold — Thu, 21 Dec 2023 09:33:01 GMT

Hi Laura,

The coordinator receives Mgmt_NWK_Update_notify commands from one of the devices (0x9970), with information about the energy levels of the different channels. The results of the energy scan shows that channel 24 has the lowest energy level, i.e. the least amount of noise. I also see the same device sending several route failures to the coordinator, but this could also be when you power cycled the device.

It seems like the Mgmt_NWK_Update_notify caused the coordinator to initiate channel change since the energy levels reported matches with the channel, and I see that the device sent the command before the coordinator sent Mgmt_NWK_Update_req to inform devices of the channel change.

You can disable processing of Mgmt_NWK_Update_notify command by calling zb_zdo_disable_network_mgmt_channel_update().
The device will still process Mgmt_NWK_Update_req commands as according to the specification.

Best regards,
Marte

RE: ZigBee radio channel change

Laura M — Wed, 20 Dec 2023 15:18:08 GMT

Hello Marte,

yes these are the logs (file name contains encryption key): devzone.nordicsemi.com/.../Nordic-automatically-sends-nwk-update-req_2800_pkt-49969_2C00_-key-8D-80-4F-1B-40-13-3D-D9-F3-E3-4B-2F-69-BD-0A-B8_2900_.pcapng

This was a stress test with:

Coordinator running our application with nRF5 SDK for Thread and Zigbee v4.1.0 and initially in channel 22. Channel mask enabling all Zigbee channel.
Nordic nRF52840 DK with radio test FW from Nordic: nRF5 SDK v17.1.0: Radio Test Example
I started TX sweep with following configurations:
- Radio mode: Ieee802154_250Kbit
- Radio TX power: 8 dBm
- Start Channel: 22
- End Channel: 23
- Time on each channel: 99 ms
5 routers joined to the network. They sometimes send Mgmt_NWK_Update_notify to notify about the state of each Zigbee channel.

Result: After many packet transmissions and after power cycling both coordinator and routers, the coordinator sent a command to move from current channel (22) to channel 24

I don't know if power cycling was needed, I did it to try to speed up the test which anyway took time.

In the logs you can see that all routers received the command to move to the new network and executed it. Unfortunately there are other routers that we integrate that do not support the nwk update req for a new channel and they wouldn't be reachable any more if coordinator changes channel.

Is there a way to disable this feature? I do not want to set the channel mask enabling only one channel but I'd like to leave the coordinator choose the best channel at the beginning, when it is installed, and keep that one. In this way I expect that in an installation with many coordinators, they will be in different channels.

Thank you

Laura

RE: ZigBee radio channel change

Marte Myrvold — Wed, 20 Dec 2023 14:36:37 GMT

[quote user="GiacomoPaci"]can you please tell us about the effect of changing nwk_manager_addr to 0x0000?[/quote][quote user="Laura M"]nwk_manager_addr instead is automatically set to 0xFFFF and I'd like to know if by changing it into deafult value (0x0000) has any effect. In particular,[/quote]

In theory, the Network Manager address should be set automatically and changing this manually without receiving Zigbee commands to change it might have unexpected consequences. However, since it does not seem like the address is set correctly in the first place you can try setting it to 0x0000 and see if it affects anything.

[quote user="GiacomoPaci"]Doing like that we will have network manager running in ZBOSS stack?[/quote]

No. Devices do not enable/disable network manager support based on the contents of the nwkManagerAddr field of NIB. This field is used for devices to know what the address of the network manager is.

[quote user="GiacomoPaci"]Do we have any setting to enable disable network manager?[/quote]

I am not aware of any API available in the application for disabling this, and I was not able to find it when looking. Is there a particular reason for why you want to disable it?

[quote user="Laura M"] I'd like to know if the bit in the screenshot you sent is automatically set.[/quote]

Yes. My screenshot was from me testing with the Zigbee Shell sample in nRF Connect SDK. I did not make any changes to the sample and configured it as a coordinator using shell commands, so the bit was set automatically.

Have you been able to collect logs from your tests?

Best regards,
Marte

RE: ZigBee radio channel change

GiacomoPaci — Wed, 20 Dec 2023 09:40:59 GMT

Marte can you please tell us about the effect of changing nwk_manager_addr to 0x0000? Doing like that we will have network manager running in ZBOSS stack?

Do we have any setting to enable disable network manager?

Can you also reply to all the other Laura's questions?

Best

Giacomo

RE: ZigBee radio channel change

Laura M — Tue, 19 Dec 2023 10:12:01 GMT

Hi Marte,

were left behind some routers (not all) and sleepy end devices connected to those routers.

Thank you for the screenshot, regarding NIB table, I see that, for instance, extended_pan_id contains the expected value, same for router_child_num and ed_child_num whose value is automatically updated when devices join the network.

nwk_manager_addr instead is automatically set to 0xFFFF and I'd like to know if by changing it into deafult value (0x0000) has any effect. In particular, I'd like to set it before calling zboss_start_no_autostart() and I'd like to know if the bit in the screenshot you sent is automatically set.

To give you some context, we call ZB_INIT() first, then we call functions to initialize nwk parameters like zb_set_network_coordinator_role and zb_set_extended_pan_id and functions to register device context (ZB_AF_REGISTER_DEVICE_CTX) and to register ep callbacks. In the end, we call zboss_start_no_autostart and we enter in while loop containing zboss_main_loop_iteration

Thank you,

Laura

RE: ZigBee radio channel change

Marte Myrvold — Tue, 19 Dec 2023 09:22:43 GMT

Hi Laura,

[quote user="Laura M"]I tried to read network manager address from NIB table by using ZB_NIB() and I saw that it is 0xFFFF in our network. Because of this, I'm afraid that maybe a third party router device sends a request to change channel.[/quote]

0xFFFF is outside the supported address range (0x0000-0xFFF7), so it seems more likely that this value is incorrect. A sniffer log should show if a channel change request is sent on the network.

[quote user="Laura M"]If I set this address to 0 (coordinator address) when network is created, what happens? Are devices being notified about Nordic being the network manager when they join the network?[/quote]

This could lead to unexpected behavior.

Devices should be informed of network manager support by the Network Manager server flag in the Node Descriptor, as in this picture:

One of your colleagues provided some more context about the issue over mail, and I have some questions based on that context to better understand the issue.

Can you confirm whether all devices are left behind or if only some of them are?
Are the devices left behind routers or end devices?
If end devices, are they sleepy or non-sleepy end devices?

Best regards,
Marte

RE: ZigBee radio channel change

Laura M — Mon, 18 Dec 2023 14:11:56 GMT

Hello Marte,

as I said we don't have the logs you are asking unfortunately but I'll try to collect them.

To have a good setup for my test I need to better understand what is not implemented related to network manager.

I tried to read network manager address from NIB table by using ZB_NIB() and I saw that it is 0xFFFF in our network. Because of this, I'm afraid that maybe a third party router device sends a request to change channel.

If I set this address to 0 (coordinator address) when network is created, what happens? Are devices being notified about Nordic being the network manager when they join the network?

thank you,

Laura

RE: ZigBee radio channel change

Marte Myrvold — Mon, 18 Dec 2023 12:21:37 GMT

Hi Laura,

Great to hear that it works on your side as well!

Regarding the underlying issue of the device automatically changing channel we have not been able to reproduce this behavior on our side. Getting logs from when this issue occurs, both application logs from the device and sniffer logs, would be helpful in figuring out what might be causing the issue.

Best regards,
Marte

RE: ZigBee radio channel change

Laura M — Fri, 15 Dec 2023 15:20:36 GMT

Hello Marte,

thank you!

I tested the code you provided and I cloud see coordinator moving to new channel. Also, new channel is retained if coordinator is rebooted.

Best,

Laura

RE: ZigBee radio channel change

Marte Myrvold — Fri, 15 Dec 2023 14:28:50 GMT

Hi,

Here is the fix that allows you to change channel on the local device. Call zb_set_channel_locally() to change the channel. This should be called from the callback function of zb_zdo_mgmt_nwk_update_req().

devzone.nordicsemi.com/.../zb_5F00_set_5F00_channel_5F00_locally.h

Best regards,
Marte

RE: ZigBee radio channel change

Laura M — Fri, 15 Dec 2023 11:59:10 GMT

Hello Marte,

we saw the issue after power cycling Nordic but it also happened at runtime (I don't know if a reboot occurred in these cases).

It isn't easy to reproduce and I don't have any logs. We are aware about channel change because our application is checking current channel every 10 minutes and it's reporting it to upper layer.

Thank you

Laura

RE: ZigBee radio channel change

Marte Myrvold — Fri, 15 Dec 2023 09:07:11 GMT

Hi Giacomo and Tim,

[quote user="GiacomoPaci"]However we experienced network channel change. Could be flash storage issue in the old nrf5 4.1.0? [/quote]

I have asked about this internally to verify.

[quote user="tcruse"]The above recommendations would allow us to control the channel switching but not prevent it, correct ?[/quote]

That is my understanding as well, and DSR are aware of the issue. However, if they were to fix this in the ZBOSS stack this would not be available in the nRF5 SDK for Thread and Zigbee, since it is deprecated and will not be upgraded.

I have forwarded your concern about this.

I will receive the hot fix for the other issue today, so hopefully I will be able to test it and share it with you later today. Although this is not the only issue you are seeing, it will provide a way to force the device to change channel, which might help with investigating a workaround for the issue you are raising here as well, but I cannot guarantee anything.

In the meantime, do you see any trend when this automatic channel change happens? Does it only occur under certain conditions or at random? Do you see anything in the application log or a sniffer log that might be relevant?

Best regards,
Marte

RE: ZigBee radio channel change

tcruse — Fri, 15 Dec 2023 05:02:06 GMT

Hi Marte,

thanks for the feedback. Giacomo and the team will look into this.

I do however have a follow up question / concern here.

Our issue is that the stack is choosing to change channels without providing the application control over this. The consequence of this, is choosing to change channels and not controlling that devices on the mesh are actually changing the channel. Resulting in devices falling off the network.
Unless I'm mis-understanding ?

The above recommendations would allow us to control the channel switching but not prevent it, correct ?

Isn't the issue different ?

Thanks

Tim

RE: ZigBee radio channel change

GiacomoPaci — Thu, 14 Dec 2023 16:25:23 GMT

Hello Marte, good new!

In the meanwhile i have another question,

so if I have understood correctly ZBOSS has not the channel manager, then if it is like this we should not experience any channel change in our network with nordic as a coordinator.

However we experienced network channel change. Could be flash storage issue in the old nrf5 4.1.0?

I saw there where flash fixies between nrf5 4.1.0 and nrf5 4.2.0.

I hope to read good news from you soon.

Best regards

Giacomo

RE: ZigBee radio channel change

Marte Myrvold — Thu, 14 Dec 2023 11:42:12 GMT

Hi Giacomo,

There was a discussion with DSR yesterday.

The feature that coordinator is automatically also the network channel manager that you refer to initially is described as "may be implemented" in the Zigbee specification, and is not mandatory. In the ZBOSS stack this is not implemented to be mandatory.
The Zigbee R22 specification does not have the best procedure regarding channel change, which is why such problems as you have reported occur. In addition, if you force a channel change and it "completes" with success, i.e. notification is sent to all devices and the coordinator changes channel as well, there is no guarantee that all devices will receive the channel change request, so some of them could be missing after this operation. For example, as is stated in the Zigbee specification, sleepy end devices will not receive the network channel change.
DSR are working on a workaround. There is a function internally in ZBOSS that can be called to make the local device change channel, but this is not available in the API. They are working on a fix to make a function with similar functionality that you can define in the application. This function should be called together with the call to zb_zdo_mgmt_nwk_update_req() which you are already using, preferably in the callback function provided by the zb_zdo_mgmt_nwk_update_req. When DSR provides this fix I will test that it works and share it with you if it does. Hopefully I will be able to share it today or tomorrow.
Please note that with the workaround some devices could still be missing after a channel change due to not receiving the channel change notification.

Best regards,
Marte

RE: ZigBee radio channel change

GiacomoPaci — Tue, 12 Dec 2023 09:21:13 GMT

Hi Marte,

Any news about the channel management issue?

We need to fix this in our application as soon as possible, do you have any suggestion to give to us as a work around?

Best regards

Giacomo

RE: ZigBee radio channel change

Marte Myrvold — Tue, 05 Dec 2023 14:23:15 GMT

Hi Tim,

The problem has been reported to DSR, the providers of the ZBOSS stack, and they are currently looking into it. I will update you as soon as I have more information.

Best regards,
Marte

RE: ZigBee radio channel change

tcruse — Tue, 05 Dec 2023 09:27:28 GMT

Hi Marte,

Thanks for looking into this.
Has there been any follow up from the developers ?

Indeed, this issue we face is pretty serious. If the Boss stack on its own is deciding to change channels, without application level intervention, whilst not controlling / verifying that the already connected devices on the mesh will follow the channel change, we have a very unstable connection.
This is a problem.

All the tests done by Laura and team to date, have pointed to the fact that the stack is indeed choosing to change the channel, and consequently leaving some devices behind, lost on the previous channels, with no way for those to reconnect successfully.

Would appreciate your support on this.

Could we organise a call with your developers and team, ASAP so we can get to the bottom of this ?

Thanks

Tim

RE: ZigBee radio channel change

Marte Myrvold — Thu, 30 Nov 2023 15:25:48 GMT

Hi Laura,

Thank you for sharing the code you used for testing this. I was able to reproduce it in nRF Connect SDK v2.5.0 as well. I have reported it to the developers and they are looking into it.

Best regards,
Marte