Zigbee Light Demo stops responding

[IanAber]

I programmed a NRF52840 Dongle with the example light switch code having changed the channel to 14 which is what my coordinator is on.

I then programmed a NRF52840DK with the example switch code and adjusted the search to look specifically for the dongle with the endpoint at 10. All the other compatible devices in my Zigbee network have endpoints set to 1.

I programmed my Smartthings hub to recognise the Nordic dongle as a simple light bulb.

I put the hub in discover mode and fire up the dongle and the DK which both connect.

The Smartthings hub recognises the dongle as a light and I can turn it on and off from the Smartthings app on my phone.

The NRF52840DK switch also finds the dongle and I can control the light on the dongle using the two buttons on the DK board. The Smartthings hub DOES NOT see the updates to the dongle made from the DK switch unless I manually refresh it, but that is another issue.

Everything is fine for a random amount of time varying between a few minutes and several hours up to just over a day on one occasion. Then the light (dongle) simply stops responding. It no longer responds to the Smartthings hub nor to the DK board switch. If I restart the switch it fails to find the light. Only when I restart the light does it work again.

Does anyone have any idea why this should be happening? Can anyone suggest how I can debug this situation?

[Jørgen Holmefjord]

Hi,

Have you tried running the light bulb application on the DK, and do you experience this issue on this as well? If you see the issue, you should enter debug mode and check if you can see where the application is stuck. You can also connect the debug out pins of the DK to SWD pinson the dongle, to debug the application running on this as well.

Best regards,
Jørgen

[Jørgen Holmefjord]

Can you please share the project you used when seeing this issue? That would help us with reproducing and debugging it.

[IanAber]

Did the keys work for you? I have been trying to do more checking myself. When I run the code on the nRF52840DK I can see that most of the time it stops in the zb_nrf52840_abort function sitting in an endless loop flushing the zb_osif_serial buffer, which is of course empty.

The stack trace shows only back to the zb_free_buf() function (1 level up) so I cannot tell where it came to from that. Is there any more I can do to enable me to help find out where it is going wrong?

[Jørgen Holmefjord]

Yes, I was able to read the sniffer trace using the keys. I have forwarded the details to our developers, hopefully they can shed some more light on what is happening from the sniffer trace and stack logs.

[IanAber]

Is there any news from development? Is there anything I can do to help? It seems like a buffer management issue since it invariably stops around the allocation or deallocation of a ZBoss buffer.

I really like the Nordic environment and the nRF52840 but I am coming under a lot of pressure to look at alternative devices from other manufacturers if I can't get this to work. All I am trying to do is to get a stable platform running in a commercially available mix of Zigbee products so I can develop five different devices to monitor and control an off grid power and air conditioning system. I have some of the code working but I can't keep the nRF52840 connected for more than a hour or so which means it is all pretty useless.

[Jørgen Holmefjord]

Unfortunately, I have not heard back from the developers yet. They are quite busy, but I have requested an update as soon as possible. Sorry for the inconvenience.

[Jørgen Holmefjord]

I just got some feedback from our developers:

Unfortunately provided zigbee_stack_trace.log does not contain any useful information. The problem is complex and hard to reproduce. However nRF5 SDK for Thread and Zigbee 3.0.0 have been made available today. This release include a lot of bug fixes.

We advise to the customer to do the following:

• build the application with the new version of the SDK
  • If the problem occurs again (we hope it won't), recompile the application with zigbee library traces:
    • use trace-enabled libraries, you will find them in $(SDK_ROOT)/sdk/nrf5/external/zboss/lib/debug
    • enable the tracing by setting in application's sdk_config.h the following:
      • ZIGBEE_TRACE_LEVEL 4
      • ZIGBEE_TRACE_MASK 0xFFFF
      • NRF_LOG_DEFAULT_LEVEL 4
      • NRF_LOG_BUFSIZE 65536
      • NRF_LOG_MSGPOOL_ELEMENT_COUNT 32
      • NRF_LOG_MSGPOOL_ELEMENT_SIZE 40
      • NRF_LOG_BACKEND_RTT_TX_RETRY_CNT 100
      • SEGGER_RTT_CONFIG_BUFFER_SIZE_UP 32768
    • Please provide source code of the application (if possible, the link http://www.cedartechnology.com/Nordic/light_bulb.zip is dead), wireshark log (or other pcap compatible) and traces.

[IanAber]

I downloaded the new SDK and tried to recompile my code. Since there appear to be many changes to the SDK and no apparent documentation explaining what must be done to existing projects to allow them to use the new 3.0 SDK I decided to simply start again with the 3.0 demo light bulb code.

I added the necessary bis to the Segger project to allow RTT log output. I am still amazed that you ship examples without this enabled! I then added all the logging stuff. I wasnt sure how to go about changing the library to the debug version so I simply replaced the two standard files with the debug versions indicated in your last post.

I changed the default Zigbee channel mask to  ZB_TRANSCEIVER_ALL_CHANNELS_MASK and added code to reset the persistent storage if the ON button is pressed on startup, This second piece should be added to the examples with a note explaining what it does. You should also make the network connection light blink when it is waiting to connect but I did not do this so I could keep as close to your original code as possible.

The program failed after only a few minutes in the same way as before.

I did not capture a new WireShark log but the old one is still on my WEB site and nothing has changed in the environment. I place a ZIP copy of the entire project and a text file with the output of the debug window. These are available at http://www.cedartechnology.com/Nordic.

[Jørgen Holmefjord]

I have forwarded your findings to the developers. They have started investigating and trying to reproduce the issue.

[Jørgen Holmefjord]

Our developers have analyzed the logs you provided, and confirm that the Zigbee stack is asserting. They know where the assert was called, but still don't know why it asserts. They have tried to reproduce the issue without success, and will now acquire SmartThings hub V2 to test with.

In the meantime, it would be of great help if you could provide information on the following points:

• Please provide as much details about the hub as you can collect (the hub name, version, revision, if possible to get also it's firmware version).
• Please provide as much information as possible about other devices in your zigbee network (do you have other routers or end devices, except light bulb demo and light switch demo?).
• Can you perform the test again and gather logs (this binary dump with <info> zboss: is perfect) AND wireshark logs from the same test run? Having both will help us correlate logs with network traffic.

[IanAber]

I have two SmartThings hubs in my environment in separate buildings. I will try the Nordic 52840DK in the second environment to see if it is truly the hub that is causing this or if it is some other device in the network. I will also try and grab a WireShark trace. It would be great if we could get some sort of time stamp on the debug log trace so we might be a ble to match it to the WireShark trace. Do you know if this is possible in Segger? My hubs are both Samsung SmartThings V2 hubs. I can provide access to my SmartThings account so you can see exactly what is in the network. I will need an email address to set up an account.

[IanAber]

I moved the nRF52840DK to my other environment which has an identical hub and it has been working for 24 hours without issue. This leads me to suspect some other device in the network is upsetting the ZBoss stack. I will move it back and run some tests by removing other Zigbee devices to try and nail down what is causing the failure. I will report back as I make progress. Thanks again for your help.