ZBOSS Stack Locking Threads

[NoahMetz]

HW/SW Versions:

nrf52840 Dongle DK

nrf Connect SDK 2.5.2

Issue:

I'm developing a Zigbee device based on the ZBOSS, using an nrf52840 dongle development kit and there seems to be an issue with the ZBOSS stack in my project configuration that I don't know how to debug(or if it's even possible without access to the source code).

Periodically(meaning seconds to hours after the stack is initialized) the device is left in a state where no other tasks are executing, and pausing with a debugger the debugging information shows that the ZBOSS thread is the only one being executed.

It's evident that no other tasks are running since the USB shell no longer responds, periodic logging over RTT in a different thread does not show over a debugger, and a test GPIO task(turning on/off a pin on a OS scheduled task) no longer changes the GPIO value.

During this time ZBOSS osif/trace doesn't produce any logs, and a device reset puts the device back into a regular state.

When sniffing the zigbee network, the issue is more likely to happen during device discovery.

The entire project configuration can be found at https://git.metznet.ca/noah/jamie, both the `prj_release.conf` and `prj_debug.conf` exhibit this behaviour(the difference between the two is that `debug` is built with additional debugging information, and `release` has FOTA enabled + uses MCUboot instead of booting the image directly).

Not sure what other logs/information would be relevant here so let me know and I can attach them. The ZBOSS osif and trace logs always show different outputs when the system is in this failure mode(using synchronous logging over RTT), and pausing with a debugger always shows the ZBOSS thread's stack in different states, then resuming causes the breakpoint on the hardfault handler to be triggered(with no relevant information in the hard fault registers, and a corrupt stack for the ZBOSS thread). The ZBOSS thread seems to be the likely cuplrit, since the stacks of all the other threads show them having yielded, with only the ZBOSS thread having a corrupt stack. Enabling asserts doesn't trigger any additional logs/faults, so it doesn't seem like the information being passed to the stack is incorrect.

Thanks for reading.

[NoahMetz]

One small clarification. The ZBOSS stack hard-faults without useful information when paused/resumed by debugger regardless of if it's in this failure mode or not, which makes collecting data on the issue difficult. I've tried putting a call to `zigbee_debug_suspend_zboss_thread` in another task/callback to freeze the stack when it gets into this failure mode, but since no other threads are executing until an external reset I haven't been successful.

[NoahMetz]

Obviously I'm going to find a sort of solution after making a public post...

In my case I've been testing after removing the USB stack completely, and I haven't run into the issue after a few network removals/rejoins/interviews.

So my working theory is that the issue I was seeing was caused by a USB thread or interrupt taking too long and breaking something in the ZBOSS stack which put it into an irrecoverable error state(while also not causing a hard fault). I will update if I notice the issue again after disabling the USB stack.

[NoahMetz]

One more update, I ran into the issue again after disabling the USB stack, so that wasn't the fix I thought it was.

I also tried tested logging via RTT in a GPIO interrupt, and it looks like that still works while the ZBOSS thread has locked up the rest of the system, so since a GPIO interrupt to write to a region of memory still works my new working theory is that the ZBOSS thread is in some kind of busy-loop where it's preventing any other OS thread from getting any CPU time, but not preventing any hardware interrupts from running.

[Maria Gilje]

Hello,

Thank you for adding your updates to the ticket.

I want to clear something up before we go further:

Are you using the nRF52840 Dongle or nRF52840 DK?

Best regards,

Maria

[NoahMetz]

Hi Maria.

This is with the nrf52840 Dongle.