Mesh back and forth seems to break connection

[kaushalyasat]

Hi,

We have one customer having two CoAP hosts and some CoAP clients in the form of wireless sensors. The sensors are paired to a single host. The pairing is actually in the app level, where the sensor discovers the network IP of the host in pairing host. All the devices have the same PANID and network key.

Recently we have seen a scenario where some sensors seemingly stopped communication with the paired host. By looking at the RSSI graphs, we thought is this caused by a sensor constantly swinging back and forth between two hosts (one host acting as a router). We dont have access to the CLI interface of the hosts as this is a remote site. We see the Sensor RSSI reported back. This is its RSSI with the router/leader immediately connected to at the time.  

Any ideas?

Cheers,

Kaushalya

[Edvin]

Hello,

Can you please try to capture a sniffer trace using the nRF Sniffer for 802154?

What do you mean by "swinging back and forth between two hosts"? Do you mean that it (an End Device, I assume) keeps changing between two routers?

Does it ever re-enter the network, or does it disconnect completely?

Do the nodes move around (physically)? Or are the nodes more or less stationary?

Best regards,

Edvin

[Christina Holman]

Are nRF Sniffer for 802154 ok? run 3

[kaushalyasat]

Hi,

We have managed to find this issue in a test system. Fortunately, we can now run sniffer. We are loosing some customers over this issue. So this is paramount we find the root cause of this.

What we have see so far is SEDs disconnecting from Leader. When we check the child table, we can see some SEDs have dropped off from it. Now we dont know what caused it. It certainly not a straight off bug, as these sensors have been working for 1-2 months. 

Out SEDs send the pol period timeout is set to 100. I think this is poll every10sec. RSSI measurement seems ok as well. Why it disconnects from the Leader for 240sec is the question.

Also another interesting observation we made was that when this disconnection happens, we lost the data from all sensors pretty much the same time. Also when we power cycle the host, all the SEDs start sending data again. This is why we think this is a host side issue. 

We are organizing a wireshark session for this. I can update more detailed picture, hopefully over the weekend.

This is a paramount issue for us now as we have lost some clients because of this. We need any help you can provide asap.

Thanks,

Kaushalya

[kaushalyasat]

Under what conditions a Leader/Router drop a child off its child table ? I thought only when no poll for the polling interval from a child and when a child gets connected via a router. But in this instance the router table is empty as well.

[kaushalyasat]

I have attached a wireshark session logged during this time. Yasin House Host Radio Test 2.pcapng 

[Edvin]

Can you see in the sniffer trace when this happens? (I can't). Also, does the logs say anything? Either from the devices dropping out or it's parents?

BR,
Edvin

[kaushalyasat]

Hi Edvin,

Unfortunately this log was captured after we saw the sensors are no longer sending data. The sensors used to work fine for about 3-4 months. So this is an extreme random and rare case. Now we have set up the debug console to the host radio and the wireshark sniffer permanently in the same place to see if we can capture when it happens.

Under what conditions the Leader/Router drop a child from its child table?

Thanks,

Kaushalya

[Edvin]

kaushalyasat said:

Under what conditions the Leader/Router drop a child from its child table?

It should only be if it stops responding, or if either the child or the parent issues a leave command for the child. I guess you are not intentionally requesting the child to leave, so my guess is that the child becomes unresponsive. Perhaps the application crashes. The logs will probably say something when it happens. You can also test this by simulating this scenario, e.g. by powering off the parent, and monitor the child node's log. 

BR,
Edvin

[kaushalyasat]

Hi Edvin,

I am trying to implement Child supervision in hope that will help. Unfortunately the documentation is not very clear to me. It is mentioned to enable 'OPENTHREAD_CONFIG_CHILD_SUPERVISION_ENABLE', which is defined in child_supervision.h in SDK. I dont think changing SDK content is a good idea as if the SDK is updated, we will loose all that. 

1. Is there a way to implement child supervision from application side?

2. What is to be done from FTD side?

3. What is to be done from SED side?

Thanks,

Kaushalya

[kaushalyasat]

Hi Edvin,

We have WDT implemented in the client, so unlikely the sensor crashes. The sensors as they are battery powered devices, have the console disabled. I think we can have special build with consoles enabled just for this.

Thanks,

Kaushalya