ZBOSS Stack Locking Threads

[NoahMetz]

HW/SW Versions:

nrf52840 Dongle DK

nrf Connect SDK 2.5.2

Issue:

I'm developing a Zigbee device based on the ZBOSS, using an nrf52840 dongle development kit and there seems to be an issue with the ZBOSS stack in my project configuration that I don't know how to debug(or if it's even possible without access to the source code).

Periodically(meaning seconds to hours after the stack is initialized) the device is left in a state where no other tasks are executing, and pausing with a debugger the debugging information shows that the ZBOSS thread is the only one being executed.

It's evident that no other tasks are running since the USB shell no longer responds, periodic logging over RTT in a different thread does not show over a debugger, and a test GPIO task(turning on/off a pin on a OS scheduled task) no longer changes the GPIO value.

During this time ZBOSS osif/trace doesn't produce any logs, and a device reset puts the device back into a regular state.

When sniffing the zigbee network, the issue is more likely to happen during device discovery.

The entire project configuration can be found at https://git.metznet.ca/noah/jamie, both the `prj_release.conf` and `prj_debug.conf` exhibit this behaviour(the difference between the two is that `debug` is built with additional debugging information, and `release` has FOTA enabled + uses MCUboot instead of booting the image directly).

Not sure what other logs/information would be relevant here so let me know and I can attach them. The ZBOSS osif and trace logs always show different outputs when the system is in this failure mode(using synchronous logging over RTT), and pausing with a debugger always shows the ZBOSS thread's stack in different states, then resuming causes the breakpoint on the hardfault handler to be triggered(with no relevant information in the hard fault registers, and a corrupt stack for the ZBOSS thread). The ZBOSS thread seems to be the likely cuplrit, since the stacks of all the other threads show them having yielded, with only the ZBOSS thread having a corrupt stack. Enabling asserts doesn't trigger any additional logs/faults, so it doesn't seem like the information being passed to the stack is incorrect.

Thanks for reading.

[Edvin]

Hello,

Maria is out of office for the next couple of weeks. I believe that was why she asked you to create a new ticket, because she knew she wouldn't be able to handle this one anymore, and therefore a recap makes the transition to another handler a bit easier. 

So let me see if I got this correctly.

You are both seeing that the application freezes, and the theory is that you run out of buffers. You can reproduce it on a dongle in the project you attached in the first post (git link) on a dongle with your Zigbee2MQTT (Z2M) coordinator. 

I don't have a dongle with a debug connector soldered onto it. Can you please try to reproduce this on an nRF52840 DK while replacing the Z2M coordinator with the network_coordinator sample?

I tried, but so far no success in letting the device connecting to the network, then pressing the reset button, which I assigned to button 1 on the DK:

buttons {
		reset_button: reset-button {
			gpios = <&gpio0 11 (GPIO_PULL_UP | GPIO_ACTIVE_LOW)>;
			label = "Zigbee Reset Button";
		};

after it was commissioned, and then repeat the process. Was this what you were doing?

Best regards,

Edvin