ZBOSS Stack Locking Threads

RE: ZBOSS Stack Locking Threads

Edvin — Tue, 25 Jun 2024 12:10:21 GMT

Hello,

Maria is out of office for the next couple of weeks. I believe that was why she asked you to create a new ticket, because she knew she wouldn't be able to handle this one anymore, and therefore a recap makes the transition to another handler a bit easier.

So let me see if I got this correctly.

You are both seeing that the application freezes, and the theory is that you run out of buffers. You can reproduce it on a dongle in the project you attached in the first post (git link) on a dongle with your Zigbee2MQTT (Z2M) coordinator.

I don't have a dongle with a debug connector soldered onto it. Can you please try to reproduce this on an nRF52840 DK while replacing the Z2M coordinator with the network_coordinator sample?

I tried, but so far no success in letting the device connecting to the network, then pressing the reset button, which I assigned to button 1 on the DK:

buttons {
		reset_button: reset-button {
			gpios = <&gpio0 11 (GPIO_PULL_UP | GPIO_ACTIVE_LOW)>;
			label = "Zigbee Reset Button";
		};

after it was commissioned, and then repeat the process. Was this what you were doing?

Best regards,

Edvin

RE: ZBOSS Stack Locking Threads

timl2415 — Wed, 19 Jun 2024 17:53:34 GMT

Agree with the above, this ticket is unresolved, recent and is likely the same issue as what I am experiencing.

I dug down further into this by adding print statements to the radio layer zboss uses in zb_nrf_transceiver.c. It seems like zboss gets into this resource hogging state when it runs out of zb buffers. I already have zb_mem_config_max.h included but I still seem to run into cases where buffers run out.

It seems like zb_trans_get_next_packet is polled until no more data is left on the radio, this api receives a new zboss buffer in a loop until all data is collected. During times of high amounts of data to rx (device interview, network start presumably due to panid conflict resolution), I've seen the number of unique buffers handed to this API reach some limit after which the stack invariably gets into the bad state.

In this state it seems like zboss cannot handle the data that is already in the packets since there was an error during rx, but it also cannot sleep since there is data to process. Funnily enough increasing the amount of net_bufs actually exacerbates this issue and more bufs end up being allocated during an rx call.

I ended up having to make a custom config with the largest amount of buffers in the buffer pool as to avoid running out ever.

RE: ZBOSS Stack Locking Threads

NoahMetz — Wed, 19 Jun 2024 13:46:53 GMT

Hi Maria,

I would appreciate if all relevant information could be contained within the ticket, including reproductions by other people.

The independent reproductions seems to confirm that this issue is present in the ZigBee stack. Is there any confirmation of a reproduction on your end?

RE: ZBOSS Stack Locking Threads

Maria Gilje — Wed, 19 Jun 2024 12:53:53 GMT

Hi Tim,

I missed your update to this ticket. If you still need support on this, please create a new ticket.

Best regards,

Maria

RE: ZBOSS Stack Locking Threads

timl2415 — Fri, 24 May 2024 20:04:15 GMT

I also configured sysview to observe the system state in this state. It seems like there are a ton of isr's in the zboss thread. Still working on figuring out what these signals are. csv attached

devzone.nordicsemi.com/.../zboss_5F00_freeze_5F00_sysview.csv

RE: ZBOSS Stack Locking Threads

timl2415 — Fri, 24 May 2024 19:56:12 GMT

Nothing particularly interesting in the sniffer trace when I run into this issue. For context my coordinator app is talking to a single smart thermostat. After boot we perform a device discovery + some interview steps. After a few boots the coordinator hangs in the zboss thread. The thermostat continues to poll data from the coordinator after it freezes.

devzone.nordicsemi.com/.../zboss_5F00_freeze_5F00_trace.pcapng

RE: ZBOSS Stack Locking Threads

timl2415 — Fri, 24 May 2024 17:53:31 GMT

I've also seen this issue crop up on both a coordinator and zigbee end device application. On the coordinator during device discovery, I've seen the zboss thread block other threads eventually triggering a wdt reset. In Ozone, halting during the "hang" always lands in a zboss API.

RE: ZBOSS Stack Locking Threads

Maria Gilje — Fri, 01 Mar 2024 15:37:49 GMT

Hello,

Sorry about the delay.

I have not been able to reproduce the issue so far.

Here are a few more things you can do:

1. Double check that the coordinator has opened the network before your device starts to rejoin.

2. See if the workaround for KRKNWK-12017: Zigbee End Device does not recover from broken rejoin procedure from the known Zigbee issues list fixes your issue.

3. Share your sniffer log (and network key) in a reply.

Best regards,

Maria

RE: ZBOSS Stack Locking Threads

NoahMetz — Wed, 28 Feb 2024 03:20:19 GMT

Have there been any updates on this?

I'm able to consistently reproduce this by calling `bdb device_reset` in the sample after changing it to use etxtended basic attributes. Since this is a blocker for any long-term zigbee device, what alternatives do I have while I await a resolution?

RE: ZBOSS Stack Locking Threads

NoahMetz — Wed, 21 Feb 2024 18:08:21 GMT

Creating a new top-level reply since I've gotten a more clear reproduction method.

Starting from the zigbee shell sample in nrf connect v2.5.2 with nrf52840dongle_nrf52840 selected as the board:

In `struct zb_device_ctx`, replace `zb_zcl_basic_attrs_t` with `zb_zcl_basic_attrs_ext_t`
Replace `ZB_ZCL_DECLARE_BASIC_ATTRIB_LIST` with `ZB_ZCL_DECLARE_BASIC_ATTRIB_LIST_EXT` and update the macro with all of the pointers to `dev_ctx.basic_attr`
Update `app_clusters_attr_init` to populate the strings with `ZB_ZCL_SET_STRING_VAL` from constant static strings, and set the other basic attributes using constant values
Boot the device in the range of a joinable network.
Wait until the device has completed the interview with the coordinator and the strings are visible on the coordinator(zigbee2mqtt in my case)
Run `bdb factory_reset` in the zigbee shell to schedule a call to `zb_bdb_reset_via_local_action` in the ZBOSS thread
Go to 5

During the 5-7 loop, step 5 will timeout and put the device into the described failure state where no threads are getting CPU time except the ZBOSS thread, and the ZBOSS thread is not processing any packets. This can be observed by the serial console not responding to any more input, and the network LED no longer showing any activity. Adding a GPIO callback to RTT log shows that hardware interrupts are still able to write to the RTT region of memory.

RE: ZBOSS Stack Locking Threads

NoahMetz — Tue, 20 Feb 2024 18:40:42 GMT

Yea the shell sample in NCS is the only one I could find that supported the dongle, so I ended up using it as an example to build off of for this application(adding four zigbee endpoints for OnOff/OnOffSwitchConfig and an endpoint for FOTA in the release build).

The sw0-sw3 switches are defined in `boards/nrf52840dongle_nrf52840.overlay` along with the apps LED and reset button assignments(since the intention is to move away from the dongle and to a custom nrf52840 board after proving the prototype). I'm intentionally not using any pre-existing pin assignments so that the code can be moved to a custom board in the future.

I was able to reproduce the same state on an "adafruit feather nrf52840"(with modifications to the dts overlay for pin assignments) after causing the device to be interviewed a few times(by calling `zb_bdb_reset_via_local_action` after a successful interview), so you should be able to reproduce the issue on any nrf52840-based development kit.

I was also able to reproduce the same state on the nrf52840 Dongle using the zigbee shell example in NCS by changing the `zb_zcl_basic_attrs_t` to a `zb_zcl_basic_attrs_ext_t` and registering/populating the fields accordingly(upon interview there was a chance for the shell to stop taking input as the ZBOSS thread took all available CPU time without completing the interview).

So to summarize(since I've spread info across a few posts now):

- The switches in the code I've shared are defined in the board overlay file

- The issue is not limited to a single SoC or bootloader since no booatloader, MCUBoot, and the NRF secure DFU bootloader(that the dongle is sold with) all have been able to reproduce this issue across two difference boards(dongle and adafruit feather)

- The issue is reproducible in the zigbee shell example when switching the basic_attrs to their extended versions, so for a minimal example that would exemplify it better.

- The issue is reproducible in the short-term case when the device is interviewed(there's a chance the interview completes successfully, and a chance the ZBOSS thread locks the OS), in the long term case I don't know what caused the failure but it could be similar read requests to the INFO endpoint with extended attributes.

RE: ZBOSS Stack Locking Threads

Maria Gilje — Tue, 20 Feb 2024 16:11:36 GMT

Thank you for the clarification. There were two reasons for needing this to be clear:

Only one of the Zigbee samples in NCS supports the nRF52840 Dongle out of the box
I saw references to more than one configurable switch in your code. The Dongle has one user configurable switch and a reset button. See the Connections and IOs part of the nRF52840 Dongle overview.

I don't have a Dongle at the moment. I will request one and see if I can reproduce the behaviour. You'll hear from me before the end of the week.

Be sure to continue to update us if you discover anything on your own.

Best regards,

Maria

RE: ZBOSS Stack Locking Threads

NoahMetz — Mon, 19 Feb 2024 16:32:20 GMT

Hi Maria.

This is with the nrf52840 Dongle.

RE: ZBOSS Stack Locking Threads

Maria Gilje — Mon, 19 Feb 2024 14:17:52 GMT

Hello,

Thank you for adding your updates to the ticket.

I want to clear something up before we go further:

Are you using the nRF52840 Dongle or nRF52840 DK?

Best regards,

Maria

RE: ZBOSS Stack Locking Threads

NoahMetz — Sun, 18 Feb 2024 21:27:53 GMT

One more update, I ran into the issue again after disabling the USB stack, so that wasn't the fix I thought it was.

I also tried tested logging via RTT in a GPIO interrupt, and it looks like that still works while the ZBOSS thread has locked up the rest of the system, so since a GPIO interrupt to write to a region of memory still works my new working theory is that the ZBOSS thread is in some kind of busy-loop where it's preventing any other OS thread from getting any CPU time, but not preventing any hardware interrupts from running.

RE: ZBOSS Stack Locking Threads

NoahMetz — Sun, 18 Feb 2024 18:34:39 GMT

Obviously I'm going to find a sort of solution after making a public post...

In my case I've been testing after removing the USB stack completely, and I haven't run into the issue after a few network removals/rejoins/interviews.

So my working theory is that the issue I was seeing was caused by a USB thread or interrupt taking too long and breaking something in the ZBOSS stack which put it into an irrecoverable error state(while also not causing a hard fault). I will update if I notice the issue again after disabling the USB stack.

RE: ZBOSS Stack Locking Threads

NoahMetz — Sun, 18 Feb 2024 17:56:55 GMT

One small clarification. The ZBOSS stack hard-faults without useful information when paused/resumed by debugger regardless of if it's in this failure mode or not, which makes collecting data on the issue difficult. I've tried putting a call to `zigbee_debug_suspend_zboss_thread` in another task/callback to freeze the stack when it gets into this failure mode, but since no other threads are executing until an external reset I haven't been successful.