Coordinator Assert during commission of 24 devices

[TKR]

nRF Connect V 1.6.0

using samples/zigbee/network_coordinator against a nRF52840, I am trying to commission a larger number of devices to join the network. Currently 24 devices, but this happens at lower numbers of 11 devices as well.

enabled the CLI - and performed the "bdb start" to begin commissioning:

uart:~$ I: Network formed successfully, start network steering (Extended PAN ID: f4ce36628124d7a9, PAN ID: 0x 4d03)
I: Device update received (short: 0xb83a, long: f4ce363045dbb779, status: 1)
I: Device update received (short: 0x7dce, long: f4ce36ddabae0f04, status: 1)
I: Device update received (short: 0x7b90, long: f4ce361d7112dd0a, status: 1)
I: Device update received (short: 0x7b90, long: f4ce361d7112dd0a, status: 1)
I: Device update received (short: 0x66df, long: f4ce36ec24ebe6a8, status: 1)
I: Device update received (short: 0xb619, long: f4ce36ec24ebe6a8, status: 1)
I: Device update received (short: 0xb619, long: f4ce36ec24ebe6a8, status: 1)
I: Device update received (short: 0x07aa, long: f4ce361054cb3284, status: 1)
I: Device update received (short: 0x07aa, long: f4ce361054cb3284, status: 1)
I: Device authorization event received (short: 0xb83a, long: f4ce363045dbb779, authorization type: 1, authori zation status: 1)
I: Child left the network (long: f4ce363045dbb779, rejoin flag: 0)
I: Device update received (short: 0x7729, long: f4ce36fde1f130fe, status: 1)
I: Device update received (short: 0x7729, long: f4ce36fde1f130fe, status: 1)
I: Device authorization event received (short: 0x7dce, long: f4ce36ddabae0f04, authorization type: 1, authori zation status: 1)
I: Child left the network (long: f4ce36ddabae0f04, rejoin flag: 0)
I: Child left the network (long: f4ce36ddabae0f04, rejoin flag: 0)
I: Device authorization event received (short: 0x7b90, long: f4ce361d7112dd0a, authorization type: 1, authori zation status: 1)
I: Unimplemented signal (signal: 50, status: 0)
I: Device authorization event received (short: 0xb619, long: f4ce36ec24ebe6a8, authorization type: 1, authori zation status: 1)
I: Unimplemented signal (signal: 50, status: 0)
I: Device authorization event received (short: 0x07aa, long: f4ce361054cb3284, authorization type: 1, authori zation status: 1)
I: Unimplemented signal (signal: 50, status: 0)
I: Device update received (short: 0xb83a, long: f4ce363045dbb779, status: 1)
I: Device update received (short: 0xb83a, long: f4ce363045dbb779, status: 1)
I: Device authorization event received (short: 0x7729, long: f4ce36fde1f130fe, authorization type: 1, authori zation status: 1)
I: Device update received (short: 0x67b0, long: f4ce36ddabae0f04, status: 1)
I: Device update received (short: 0x67b0, long: f4ce36ddabae0f04, status: 1)
I: Device authorization event received (short: 0xb83a, long: f4ce363045dbb779, authorization type: 1, authori zation status: 1)
I: Device authorization event received (short: 0x67b0, long: f4ce36ddabae0f04, authorization type: 1, authori zation status: 1)

some devices begin to join, then an assert happens at: 

uart:~$ E: Fatal error occurred
ASSERTION FAIL @ WEST_TOPDIR/nrfxlib/nrf_802154/driver/src/nrf_802154_notification_swi.c:154

Call stack is this:

We a trying to prepare this product for production and this is a major issue at the moment.  Please advise the best way to enable the maximum devices to join the network.

[Marjeris Romero]

Hi,

I am so sorry for the late reply. I have forwarded your questions about the maxiomum devices that can be added to a network to our Zigbee team but haven't gotten an answer yet. In the old nRF5 SDK for Thread and Zigbee we had conducted tests with up to 24 devices, but with no architectural limitation for networks larger than that. The practical limiting factor will be the Trust Center memory (in the coordinator device) as the TC must hold link keys to all connecting devices.

Do you have a sniffer trace that can help us debug when the devices are joining your coordinator network? I will like to see what is happening on air. In the log output it looks like some devices are being added multiple times.

Best regards,

Marjeris

[TKR]

Hi Marjeris,

To be clear, this seems to be immediately evident if you try to join all devices at one time. If we stagger the endpoints that come in, then the problem is rarely, if ever occurring.

Best regards,

TKR

[Marjeris Romero]

Hi Larry,

I have passed this information to our Zigbee team to get some help at decoding the error messages, I am sorry for the delays but I will am waiting for their feedback about this and on how to change the number of ZDO transactions limit.

[Marjeris Romero]

Hi Larry,

The limit is set at the ZBOSS compile time (CONFIG_ZB_ZDO_TRAN_TABLE_SIZE) and cannot be changed by the application. We don't have the possibility to recompile the ZBOSS stack we delivered, since it will then not be a certified stack for instance. So I am afraid this is not a configurable limit.

Our developers are asking if you could log all values returned by the zb_osif_nvram_write? It is possible that the assert(122,1892) is caused by the nvram write error at the OSIF layer.

Best regards,

Marjeris

[Larry Martin]

We have not seen any zb_osif_nvram_write errors, either before the assertions or at any other time.

[Larry Martin]

FYI, we just tried NCS 1.7.0 with zboss 3.8 and see similar asserts during commissioning of several nodes at once.

[Marjeris Romero]

Hi Larry,

The file 122 is the zb_nvram.c and the line 1892 is inside zb_nvram_write_dataset(..).

We really would like to get a dump of all NVRAM logs & operations (from zb_nrf_nvram.c) to understand which operation causes the function to fail.