Coordinator Assert during commission of 24 devices

RE: Coordinator Assert during commission of 24 devices

Larry Martin — Wed, 03 Nov 2021 18:45:24 GMT

Changing ZB_CONFIG_SCHEDULER_Q_SIZE from 48 to 127 changed the behavior of the problem. I've been testing with configuration 1 above and now when trying to "gang discover" 20+ devices it now seems that zboss runs out of buffers and never recovers (zb_buf_memory_low() returns 1 until we reset). Initial testing with configurations 2 and 3 suggests we may just be trading off the likelihood of a zboss assertion with this unrecoverable low memory condition.

RE: Coordinator Assert during commission of 24 devices

sedr — Wed, 03 Nov 2021 14:45:28 GMT

Hi Larry,

Could you try three configurations below and provide feedback whether or not any fail is observed during commissioning? (For coordinator device)

1.
    CONFIG_NRF_802154_RX_BUFFERS set to 64
    ZB_CONFIG_IOBUF_POOL_SIZE defined to 127U
    ZB_CONFIG_SCHEDULER_Q_SIZE defined to 127U

2.

    CONFIG_NRF_802154_RX_BUFFERS set to 32
    ZB_CONFIG_IOBUF_POOL_SIZE defined to 127U
    ZB_CONFIG_SCHEDULER_Q_SIZE defined to 127U

3.

    CONFIG_NRF_802154_RX_BUFFERS set to 16
    ZB_CONFIG_IOBUF_POOL_SIZE defined to 127U
    ZB_CONFIG_SCHEDULER_Q_SIZE defined to 127U

RE: Coordinator Assert during commission of 24 devices

Marjeris Romero — Tue, 02 Nov 2021 09:47:58 GMT

Hi Larry,

The file 122 is the zb_nvram.c and the line 1892 is inside zb_nvram_write_dataset(..).

We really would like to get a dump of all NVRAM logs & operations (from zb_nrf_nvram.c) to understand which operation causes the function to fail.

RE: Coordinator Assert during commission of 24 devices

Larry Martin — Mon, 27 Sep 2021 17:23:31 GMT

FYI, we just tried NCS 1.7.0 with zboss 3.8 and see similar asserts during commissioning of several nodes at once.

RE: Coordinator Assert during commission of 24 devices

Larry Martin — Sat, 25 Sep 2021 16:17:54 GMT

We have not seen any zb_osif_nvram_write errors, either before the assertions or at any other time.

RE: Coordinator Assert during commission of 24 devices

Marjeris Romero — Mon, 20 Sep 2021 11:34:29 GMT

Hi Larry,

The limit is set at the ZBOSS compile time (CONFIG_ZB_ZDO_TRAN_TABLE_SIZE) and cannot be changed by the application. We don't have the possibility to recompile the ZBOSS stack we delivered, since it will then not be a certified stack for instance. So I am afraid this is not a configurable limit.

Our developers are asking if you could log all values returned by the zb_osif_nvram_write? It is possible that the assert(122,1892) is caused by the nvram write error at the OSIF layer.

Best regards,

Marjeris

RE: Coordinator Assert during commission of 24 devices

Marjeris Romero — Thu, 16 Sep 2021 09:19:39 GMT

Hi Larry,

I have passed this information to our Zigbee team to get some help at decoding the error messages, I am sorry for the delays but I will am waiting for their feedback about this and on how to change the number of ZDO transactions limit.

RE: Coordinator Assert during commission of 24 devices

Larry Martin — Tue, 14 Sep 2021 12:48:28 GMT

Yes, these assertions are in the coordinator.

We restrict our code to a maximum of 3 pending transactions at any one time. We originally added this limit because we discovered that zboss was running out of buffers (zb_buf_memory_low() returning true) during commissioning of multiple devices. Enforcing a limit of 3 pending transactions helped reduce the number of zboss assertions. We eventually figured out how to increase the number of zboss buffers using a custom zb_mem_config header file setting ZB_CONFIG_IOBUF_POOL_SIZE to 127U, and since then we haven't seen the low memory condition. We continue to see the assertions during commissioning and leave operations.

We use zdo_mgt_leave_req() to issue the leave request, and consider the transaction pending until the callback. As noted above, we allow a maximum of 3 pending transactions of any type, including leave.

I am curious about the limit on the number of ZDO transactions. What it the source of this limit? Is it configurable?

And thanks for your help!

Update: I tried reducing the number of pending transactions to 1 at a time and still got the zdo assert(112,1892). As we ask devices to leave I see that we're getting a large number of ZB_ZDO_SIGNAL_DEVICE_UPDATE signals (status=device left). I also see that we get multiple ZB_ZDO_SIGNAL_DEVICE_UPDATE signals for each device as it leaves, maybe one per neighbor? (Each node in our network is configured as a router+endpoint.) Perhaps all this traffic is what's triggering the assertion?

Here's an extract from our log showing the multiple device_update signals after a leave:

{cmd:leave,addr:0x494E,device_addr:0x0000000000000000,remove_children:0,rejoin:0} <-- This calls zdo_mgmt_leave_req(...)
{rsp:"leave",tsn:203}
{nfy:"throttle",can_send:0,pending:1,max:1,mem_low:0}
{nfy:"leave",tsn:203,status:0x0} <-- sent from the zdo_mgmt_leave_req(...) callback
{nfy:"throttle",can_send:1,pending:0,max:1,mem_low:0}
{cmd:leave,addr:0x23A2,device_addr:0x0000000000000000,remove_children:0,rejoin:0} <--another call to zdo_mgmt_leave_req(...)
{rsp:"leave",tsn:204}
{nfy:"throttle",can_send:0,pending:1,max:1,mem_low:0}
{nfy:"device_leave",addr:0xf4ce3683c5e670ce,rejoin:0} <-- ZB_ZDO_SIGNAL_LEAVE for 0x494E above
{nfy:"device_update",addr:0x494e,status:2} <-- ZB_ZDO_SIGNAL_DEVICE_UPDATE, status = device left
{nfy:"device_update",addr:0x494e,status:2} <-- multiple signals received, possibly from multiple routes???
{nfy:"device_update",addr:0x494e,status:2}
{nfy:"device_update",addr:0x494e,status:2}
{nfy:"device_update",addr:0x494e,status:2}
{nfy:"device_update",addr:0x494e,status:2}
{nfy:"leave",tsn:204,status:0x0}

RE: Coordinator Assert during commission of 24 devices

Marjeris Romero — Tue, 14 Sep 2021 09:56:29 GMT

Hi,

Are you trying to send 24 ZDO leave frames at once? There is a hard limit on the maximum number of concurrent ZDO transactions (currently set to 16).

The assert is happening on the coordinator side right?

RE: Coordinator Assert during commission of 24 devices

Larry Martin — Mon, 13 Sep 2021 14:47:37 GMT

FYI, here are the assertions we've seen after increasing the RX buffers:

zdo assert(122,1892)
zdo assert(112,1892) <-- different file, 112 vs 122
zdo assert(124,589)
zdo assert(124,635)

Assertions seem to occur during high traffic bursts.

RE: Coordinator Assert during commission of 24 devices

TKR — Sun, 12 Sep 2021 21:42:07 GMT

Hello Marjeris,

By increasing the buffers to 64, it did become a lot more stable so that was a good suggestion. We are getting boss zdo asserts when we ask all devices to leave, though. If we cannot find a work around I'll post another ticket with details.

TKR

RE: Coordinator Assert during commission of 24 devices

Marjeris Romero — Fri, 10 Sep 2021 08:50:44 GMT

Hi TKR,

We have two theories:

From the problem description it sounds very likely to be a problem caused by packet collissions due to high traffic, but there should be ways to handle this according to our Zigbee team.

It could also be an issue with lack of RX buffers. You may try to increase the number of buffer via Kconfig first and see if that helps:

# Increase the number of RX buffers
CONFIG_NRF_802154_RX_BUFFERS=32

Try this and let me know how it goes. I will update you when I have more information as we continue to investigate this internally.

Best regards,

Marjeris

RE: Coordinator Assert during commission of 24 devices

TKR — Wed, 08 Sep 2021 20:02:39 GMT

Hi Marjeris,

To be clear, this seems to be immediately evident if you try to join all devices at one time. If we stagger the endpoints that come in, then the problem is rarely, if ever occurring.

Best regards,

TKR

RE: Coordinator Assert during commission of 24 devices

Marjeris Romero — Mon, 06 Sep 2021 20:57:31 GMT

Hi,

I am so sorry for the late reply. I have forwarded your questions about the maxiomum devices that can be added to a network to our Zigbee team but haven't gotten an answer yet. In the old nRF5 SDK for Thread and Zigbee we had conducted tests with up to 24 devices, but with no architectural limitation for networks larger than that. The practical limiting factor will be the Trust Center memory (in the coordinator device) as the TC must hold link keys to all connecting devices.

Do you have a sniffer trace that can help us debug when the devices are joining your coordinator network? I will like to see what is happening on air. In the log output it looks like some devices are being added multiple times.

Best regards,

Marjeris