Mesh back and forth seems to break connection

RE: Mesh back and forth seems to break connection

kaushalyasat — Thu, 20 Jun 2024 01:07:55 GMT

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/489524"] Are you saying that you have two networks that each router will be part of? One small network between itself and the sensors, and one other "big" network with all the routers and the gateway?[/quote]

Not exactly. We need a router per installation. This installation is like a home. Then the sensors in that property shall be commissioned to that router. If certain sensors are out of reach, we provision a range extender (another router) to bridge the gap. All in all, the entire property shall have a single network.

The neighboring property, say this is a multi-storey apartment building, will have another network on their own as I mentioned before. So each property will have a small network of their own. This way a sensor of one property shall not mesh to a router in another property, which could yield to partitioning.

Still a partition could happen inside a property due to some radio disturbance later on. All the property owner need to do is

1. power cycle each sensor that has disconnected.

2. or worst case power cycle the router/routers.

Also as we report partitionID of each router to a dashboard, we can remotely detect that there is partitioning inside a property, as all the routers that belong to a property are grouped together. Only primary router will be connected to internet and if we cant see other routers, that means they are partitioned.

I have had a discussion with OpenThread GIT group and their opinion is the partitioning and the SEDs behaviors are as part of the OpenThread spec. But I think this behavior is not correct as it can yield to non-recoverable communication failures in the network. Even if we detect partitioning in the SEDs, there is nothing application level program can do to connect to a different parent.

Only way I can think of is the routers to detect partitioning and do a individual network restart until a single partition is made. If after couple of cycles still not resolvable then the primary router should report this to the user and the dashboard. Then human intervention is needed to commission another router to resolve.

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

Edvin — Wed, 19 Jun 2024 12:59:20 GMT

If you don't have a stable Thread network, I am not sure whether Openthread is really the best protocol, but I guess it is a bit late to change this now?

1: I think that sounds like a good idea, as it would make sure that your entire network is in one partition. However, it seems like this is not 100% going to work, since most of the time, you get all messages as expected, but after some months, the network suddenly splits. Perhaps caused by some added radio noise in the area. This means that this may not be detected during installation. But it could perhaps be used to detect the partitioning. I don't know whether both partitions have internet access, and hence, whether they can report that they are on a separate partition. You are more familiar with the setup, and can say whether this would work or not.

2: Are you saying that you have two networks that each router will be part of? One small network between itself and the sensors, and one other "big" network with all the routers and the gateway? I am not sure how this would solve it if the big network struggles with partitioning. Then you will still not be able to reach all the sensors from the small networks.

If the issue is that there is no 100% stable route between the sensor and the gateway, then adding more routers seems like adding more routers is the way to go. The key is to understand when this is the case, and your proposal #1 would perhaps help with this?

Best regards,

Edvin

RE: Mesh back and forth seems to break connection

kaushalyasat — Mon, 17 Jun 2024 23:59:02 GMT

Hi Edvin,

The issue we see here is actually not partitioning, but the SEDs cant seem to proactively dislodge from a router and do a MLE reattach to another. There is no guarantee that the SED will not be connected to the same router again. From the Thread standpoint this may be a perfectly valid scenario, but its a killer or non-recoverable error for us.

Does 'Search for a better parent' feature pick up a new router ?

Now we are implementing two solutions.

1. Make the partitionID available in our dashboard for each router. It the commissioning stage we make sure all routers form a single partition. If not we introduce additional routers until we can get a single partition.

2. Instead of a single big network, we implement networks per router. We have one main router in a installation and if we need to bring in more, we commission them with main router, so that they all have same network credentials as the main router. Similarly the SEDs will be commissioned in. At the point of commissioning we will have a single contagious network per installation. This way, meshing will be limited within that network.

The only other issue this doesn't solve is a one case a test user reported, where he says he didn't have no more than one router in his system at all times, but he saw his sensors disconnecting. As this is only single case we have reported, we will park this scenario till we have more conclusive data.

What do you think?

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

Edvin — Mon, 17 Jun 2024 13:45:39 GMT

Hello Kaushalya,

Interesting. If the parents split into two different networks, then there will exist two different networks with the same credentials, not able to communicate with one another. And it is not possible to connect the two networks using a SED device.

The solution is "simple". You need to make sure that if you struggle with that the two partitions of the network can't reach one another, you need to add another router that can reach both the network partitions (so that they are no longer separate partitions).

I believe this is where we started this conversation. I remember asking for state changes to see if a router became leader (which it will if part of the network splits out in it's own partition).

Is it an option to add another router making sure the network doesn't split into two partitions?

Best regards,

Edvin

RE: Mesh back and forth seems to break connection

kaushalyasat — Thu, 13 Jun 2024 23:53:02 GMT

Hi Edvin,

Sorry about my delay, I was not well past couple of days and only now back in work.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/487218"]But the supervision messages are only between the child and it's parent, so this message is not routed to any other routers than the parent itself. Right?[/quote]

No the message is send to another router. The parent is just routing it through.

I think we have identified a potential pitfall in openthread networks. That is partitioning. We think our failure mechanism can be explained by that.

Think of two routers and a set of sensors (SEDs). If both routers are in the same partition, then no issues. But for whatever reason one router lost its connection to the other, then there will be two network partitions. The problem happens when the Sensors can still reach both routers. If a sensor that is sensing data the router1 suddenly lost its connection and connected to router2, the data path will be broken, but the Sensor will happily stay in the current connection until another MLE session decide to connect back to router1. The issue is for the MLE protocol, every router in range is a potential parent, but this may not work for application data.

It looks like we cant constrain the MLE process to limit the parent search to its own partition. We can implement a two way handshake from application level and initiate another MLE session from the Sensor, but there is no guarantee that it may not choose a parent from a different partition.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/487218"]So this is the log from the router. When did it crash? (timestamp?)[/quote]

Sorry Edvin, as this happened when we were not at office, we dont know exactly at what time this happened. So it may be very diff to extract it from the log.

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

Edvin — Tue, 04 Jun 2024 07:31:14 GMT

[quote user="kaushalyasat"]

From what we have seen, its not the node dropping off. The nodes (SEDs) are sending their data out to the associated parent. I think the issue is the parent gets into some error state all of a sudden and stops sending these packets any further.

[/quote]

But the supervision messages are only between the child and it's parent, so this message is not routed to any other routers than the parent itself. Right?

[quote user="kaushalyasat"]

What puzzles me is how long it took for the problematic router to be healed. Also this 'healing' happened when I connect to the console, so dont know if that had any effect. As this is a rare event, very difficult to deep diagnose.

[/quote]

How long did it take?

[quote user="kaushalyasat"]

This is what I saw from the log while I was connecting to router 0x9400.

[/quote]

So this is the log from the router. When did it crash? (timestamp?)

[quote user="kaushalyasat"]I guess if I wait 240 sec timeout, and try thread stop/start, I might get connected to a new router. What do you think?[/quote]

I would assume it does. But I guess the main issue here is that this is not happening at all times, right?

RE: Mesh back and forth seems to break connection

kaushalyasat — Fri, 31 May 2024 00:05:20 GMT

Hi Edvin,

Sorry I forgot to attach the pcap log.

devzone.nordicsemi.com/.../5633.May_2D00_30_2D00_prov-success-and-fail.pcapng

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/486592"]Child supervision will not prevent nodes from dropping out of the network. It will just make it easier to detect whether it has happened or not.[/quote]

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/486592"]I would say that this is the expected behavior (from all other than the router that suddenly disconnected), don't you agree?[/quote]

This is what I saw from the log while I was connecting to router 0x9400.

devzone.nordicsemi.com/.../7181.0x9400-log.txt

Also, is there a way to detach from a parent and start a renegotiate to connect to a different router? I tried 'ot detach', 'ot thread stop' and 'ot thread start' from the SED. But I think since the SED is still in the child table of the first parent, all that happens is that I get a new RLOC. I guess if I wait 240 sec timeout, and try thread stop/start, I might get connected to a new router. What do you think?

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

Edvin — Thu, 30 May 2024 08:50:52 GMT

Hello Kaushalya,

[quote user="kaushalyasat"]Normally when it powers up it connects to the network and change it's state to a router/leader. But this time it stayed on as a child![/quote]

As you can see in the Openthread documentation, the network will strive to keep the number of routers below 15, if it is feasible. This is in order to be able to expand the network in any direction (physically) when needed. Therefore, if there are no devices that need to attach to your new device, and you already have a large amount of routers, there is no need to promote that device to a router.

[quote user="kaushalyasat"]I guess a FTD will remain a child as long as no child connects to it, am I correct?[/quote]

Yes, as long as there is a decent coverage of routers in the area of the new child.

[quote user="kaushalyasat"]

I can see that in 'coap_send_request (...)' function, it is hard coded to be non-ack type (COAP_TYPE_NON_CON). I dont know if there is a way to request with confirmation using any API.

Suppose I clone this 'coap_send_request (...)' and modify it to send ACK packet, then I would need to pass a call back with it to know the ack is successful or not?

[/quote]

If there are too few routers in the network, it will promote some of them to routers, so that the network is ready to accept new child nodes.

[quote user="kaushalyasat"]This is possible. Our understanding is even it takes time to renegotiate new paths, it should be ok as we are sending temperature info in every 30 sec. Even if the rerout takes couple of minutes, it should be ok. But the issue is once the sensor's (SED) data is lost, it remains so for days.[/quote]

Child supervision will not prevent nodes from dropping out of the network. It will just make it easier to detect whether it has happened or not.

...

I am surprised that it doesn't take into account whether you provide a callback function or not, and that you could specify whether the coap_send_request() should be acked or not. But you are right, you can clone this function, and tell it to use COAP_TYPE_CON. Then you can also provide a coap_reply_t reply_cb, which will trigger when the callback occurs. I believe the node will automatically retransmit messages that are not acked, but you can use this while debugging.

...

I agree that it shouldn't happen. If the connection to the parent is lost (because it is powered off), then the child should try to reconnect to the network through some other node.

[quote user="kaushalyasat"]After a while it 'auto-healed' by itself and my provisioning packets are now received by the SED.[/quote]

I would say that this is the expected behavior (from all other than the router that suddenly disconnected), don't you agree?

You don't happen to have any logs from the router that suddenly greyed out?

Best regards,

Edvin

RE: Mesh back and forth seems to break connection

kaushalyasat — Thu, 30 May 2024 05:52:42 GMT

Hi Edvin,

Dont know if this is relevant for this issue, but today I saw a parent router going into some error state. I was provisioning and un-provisioning an SED from a FTD for a new fw feature and all of a sudden I noticed the prov reply send by the FTD does not reach the SED. When I look into wireshark log, I could see the parent router not sending the reply back. Also when I had a look at TTM, it showed this parent router grayed out and ext address field was empty. After a while it 'auto-healed' by itself and my provisioning packets are now received by the SED. I wonder if this could be the error state the router ends up when we cant get SED data.

What do you think?

Following is a pcap of when the provisioning was working and stopped. Provisioning router is 0xb000, provisioning SED is 0x9423 and the parent router 0x9400

devzone.nordicsemi.com/.../May_2D00_30_2D00_prov-success-and-fail.pcapng

I have also created a separate thread to show the TTM findings. An OpenThread router shown as grayed out in TTM

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

kaushalyasat — Thu, 30 May 2024 01:05:15 GMT

Hi Edvin,

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/485822"]Is it possible to add some logging here, to see if there is a state change? It doesn't necessarily change to detached, so printing the actual state change can be helpful, if this is the case.[/quote]

Something interesting I saw couple of days ago with a FTD on my desk. Normally when it powers up it connects to the network and change it's state to a router/leader. But this time it stayed on as a child! Now if a router, who is actively routing packets to another router suddenly change state to a child, I guess we can explain the behavior we are discussing here.

I guess a FTD will remain a child as long as no child connects to it, am I correct? But on the other hand I have seen many instances where a router's child table is empty, but it remains as a router. What conditions determine this transition from child-router and vice versa?

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/485822"]I don't understand this question. And I don't understand how child supervision is relevant if the packet is lost between two nodes that are not children.[/quote]

Sorry, what I was trying to get at is, we implemented child supervision to prevent a SED disconnection from a destination router, but may be not a solution here.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/485822"] Is it a message that is supposed to be acknowledged or not?[/quote]

I can see that in 'coap_send_request (...)' function, it is hard coded to be non-ack type (COAP_TYPE_NON_CON). I dont know if there is a way to request with confirmation using any API.

Suppose I clone this 'coap_send_request (...)' and modify it to send ACK packet, then I would need to pass a call back with it to know the ack is successful or not?

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/485822"]Could it be that the packet path between the source and destination was through a router that was suddenly powered off?[/quote]

This is possible. Our understanding is even it takes time to renegotiate new paths, it should be ok as we are sending temperature info in every 30 sec. Even if the rerout takes couple of minutes, it should be ok. But the issue is once the sensor's (SED) data is lost, it remains so for days.

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

Edvin — Fri, 24 May 2024 12:47:34 GMT

[quote user="kaushalyasat"]But if we remove child sensor, we loose the data packet origination. So after that the parent and next routers wouldn't get any data packets anyway. So the behavior is not same isn't it, though the end result is the same? [/quote]

Of course, but I am saying that if you were to generate dummy data at the parent and send it to the destination, in theory we could see the same, because the packets are lost between the parent router and the destination router. I am just thinking of possible ways to easier reproduce the issue.

[quote user="kaushalyasat"]We don't do anything special here. Also in the production hosts, we dont populate this LED. [/quote]

Is it possible to add some logging here, to see if there is a state change? It doesn't necessarily change to detached, so printing the actual state change can be helpful, if this is the case.

[quote user="kaushalyasat"]Q1. If both routers support child supervision, does the supervision packet originate from the destination router or the parent router?[/quote]

I don't understand this question. And I don't understand how child supervision is relevant if the packet is lost between two nodes that are not children.

Q2: Depends. Is it a message that is supposed to be acknowledged or not? It wouldn't know that the scenario 2 happened, but it can detect a missing acknowledgement. The routers should be able to detect it, though. Because all messages should be acknowledged in the 802.15.4 layer.

Q3: Could it be that the packet path between the source and destination was through a router that was suddenly powered off? I am not 100% sure of the details, but openthread is not known for being very fast at determining that nodes are gone, and enabling new routes.

Best regards,

Edvin

RE: Mesh back and forth seems to break connection

kaushalyasat — Thu, 23 May 2024 00:47:51 GMT

Hi Edvin,

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/485358"]So in theory, if we take out the child sensors, and the two routers send messages back and forth, we could see the same behavior? Do you agree? [/quote]

What you mean is like shutdown the child sensor? But if we remove child sensor, we loose the data packet origination. So after that the parent and next routers wouldn't get any data packets anyway. So the behavior is not same isn't it, though the end result is the same? Moreover, in this instant, the child sensor is successfully sending the packets.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/485358"]Do you have a state changed callback in your parent's application?[/quote]

Yes we have, as our application is derived of the same example. This is what we have.

static void on_thread_state_changed(uint32_t flags, void *context)
{
	struct openthread_context *ot_context = context;

	if (flags & OT_CHANGED_THREAD_ROLE) {
		switch (otThreadGetDeviceRole(ot_context->instance)) {
		case OT_DEVICE_ROLE_CHILD:
		case OT_DEVICE_ROLE_ROUTER:
		case OT_DEVICE_ROLE_LEADER:
			dk_set_led_on(OT_CONNECTION_LED);
			break;

		case OT_DEVICE_ROLE_DISABLED:
		case OT_DEVICE_ROLE_DETACHED:
		default:
			dk_set_led_off(OT_CONNECTION_LED);
			deactivate_provisionig();
			break;
		}
	}
}

We don't do anything special here. Also in the production hosts, we dont populate this LED.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/485358"]Does that trigger when the nodes fall out?[/quote]

I am afraid we don't know. As we only see this issue once it happened. What you suggest is role change from 'router' to 'disabled' kind of? I will check the role of the host once I detect this again. I will populate the LED mentioned above as well.

I am trying to understand what the MLE does in the following scenario.

1. sensor -> parent router -> destination router - good

2. sensor -> parent router -x- destination router - bad

Q1. If both routers support child supervision, does the supervision packet originate from the destination router or the parent router?

Q2. Sensor wouldn't know scenario 2 has happened?

Q3. I guess when scenario 2 happens parent router would try to find another path via MLE? In our case we have about 10 routers all over the lab. One or two may be power cycled or shut down but not all and they are never moved from one place to the other. Also when I shutdown all the other routers, the sensors all connected back to the destination router. That proves there was at least one path to the parent router. (These sensors were about 10-50cm away from the parent router)

Cheers,

Kaushalya

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

Edvin — Wed, 22 May 2024 10:30:38 GMT

Hello Kaushalya,

So in theory, if we take out the child sensors, and the two routers send messages back and forth, we could see the same behavior? Do you agree?

Do you have a state changed callback in your parent's application?

If you look at e.g. the ncs\nrf\samples\openthread\coap_server sample, in coap_server.c, you can see the on_thread_state_changed() callback.

Does that trigger when the nodes fall out?

BR,

Edvin

RE: Mesh back and forth seems to break connection

kaushalyasat — Wed, 22 May 2024 02:19:09 GMT

Hi Edvin,

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/485225"]Ok, so even when you do not see that the messages are being received, you can see that it prints the return value (+)47, which is the length of the packet that you are sending?[/quote]

Yes. Yesterday we had the first sensor fall off 'with child supervision'. I couldn't see any messages regarding child supervision though in these sensors. But I can verify the following.

1. sensors are transmitting and the data is sent to its parent successfully. After the parent, I couldn't trace the packet any more as they cannot be decrypted.

2. intended destination doesnt receive these packets in application level

So I think the issue is in loosing a FTD to FTD (router to router) connection in multi-hop scenarios.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/485225"]So this means that so far you have not confirmed that the tx function returns 47 while the issue is ongoing? Or have you confirmed this?[/quote]

I can confirm that 47 is received while a sensor or more precisely the system in this state. As I mentioned, I dont think this is relevant to the sensor (SED). This may well be an issue in router to router hop.

I am now researching how the MLE works. What happens when a router looses its connection with another router? I think it will try to find another path. Now my question is why a path cannot be found to the destination router/leader, where it was established earlier? Only change would happen would be some routers may be power cycles/off. But I have verified that the sensors could connect to the destination even with all the other routers are powered down.

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

Edvin — Tue, 21 May 2024 20:31:25 GMT

[quote user="kaushalyasat"]Yeah, we always see 47 so far.[/quote]

Ok, so even when you do not see that the messages are being received, you can see that it prints the return value (+)47, which is the length of the packet that you are sending?

[quote user="kaushalyasat"]Yes I can see the LOG_ERR, from both my application level and also from coap_send_request(). I tested it from console and also RTT viewer.[/quote]

Ok, good. That means that there is not some sort of config that disables that logging instance.

[quote user="kaushalyasat"]Also in this section we maintain counters for failed tx and successful tx. So far we havent seen any failed, but again it might take months before that happen.[/quote]

So this means that so far you have not confirmed that the tx function returns 47 while the issue is ongoing? Or have you confirmed this?

Best regards,

Edvin

RE: Mesh back and forth seems to break connection

kaushalyasat — Tue, 14 May 2024 01:02:50 GMT

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/482993"]If you see error messages printed from that file in general. You can test with adding "LOG_ERR("Test");", to see if these error messages are visible in the log at all.[/quote]

Yes I can see the LOG_ERR, from both my application level and also from coap_send_request(). I tested it from console and also RTT viewer.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/482993"]Where are those from? What trigger these?[/quote]

This log message is sent by send_sensor_update () in coap_client_utils.c, just before calling the coap_send_request (). So we know the flow is working till that point.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/482993"]It would be more interresting to continuously see the return value from coap_send_message()[/quote]

Agree. Unfortunately in fw Rev 1.1.1.0, which I sent to you first, doesnt have that - my bad. In latest fw it shows it and also we have implemented a noinit memory section where we keep the last returned value to coap_send_request(). Also in this section we maintain counters for failed tx and successful tx. So far we havent seen any failed, but again it might take months before that happen.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/482993"]Try adding prints of the return value of the function that doesn't work (regardless of whether it is 0 or something else). [/quote]

It is done in the latest fw. We are waiting for any sensor to go into this mode again. Currently we get 47 as the return value, which I think the number of bytes send(?)

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/482993"] Print that it returns 0 even though it is disconnected?[/quote]

It prints whatever returned from coap_send_request() as follows. We havent seen returning 0 as there is always a network to connect to in the lab. Also if it is not connected, the send_sensor_update () wouldn't get called.

int ret;
.
.
.
ret = coap_send_request(COAP_METHOD_PUT, (const struct sockaddr *)&unique_local_addr, sensor_option, payload, sizeof(payload), NULL);
LOG_INF ("ZS %d, RSSI %d, LQI %d, LQO %d, FW %04x RET: %d, RLOC: %04x", the_sensor_device->zoneState, RSSI, linkQalIn, linkQualOut, FWRevNum, ret, rloc);

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/482993"]stop printing alltogether?[/quote]

What you mean is the printing suddenly stop without any reason? We havent seen anything like that.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/482993"]Print that it returns something else than 0?[/quote]

Yeah, we always see 47 so far.

RE: Mesh back and forth seems to break connection

Edvin — Mon, 13 May 2024 10:55:01 GMT

[quote user="kaushalyasat"]I was referring to the error logs as marked above. I dont see any of these errors in my case. So I assume that 'coap_send_request ()' executes without any error. Am I correct?[/quote]

If you see error messages printed from that file in general. You can test with adding "LOG_ERR("Test");", to see if these error messages are visible in the log at all.

[quote user="kaushalyasat"]I dont this this is the case as I can see this log message continuously from a disconnected SED. [/quote]

Where are those from? What trigger these?

It would be more interresting to continuously see the return value from coap_send_message(), or whatever message you use to send, at the time of the disconnection?

Try adding prints of the return value of the function that doesn't work (regardless of whether it is 0 or something else).

Does it:

1: Print that it returns 0 even though it is disconnected?

2: stop printing alltogether?

3: Print that it returns something else than 0?

BR,

Edvin

RE: Mesh back and forth seems to break connection

kaushalyasat — Mon, 29 Apr 2024 01:26:26 GMT

Hi Edvin,

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/480697"]I was thinking about these. What error logs do you refer to?[/quote]

int coap_send_request(enum coap_method method, 
					  const struct sockaddr *addr,
		      		  const char *const *uri_path_options, 
					  uint8_t *payload,
		      		  uint16_t payload_size, 
					  coap_reply_t reply_cb)
{
	int ret;
	struct coap_packet request;
	uint8_t buf[MAX_COAP_MSG_LEN];

	ret = coap_init_request(method, COAP_TYPE_NON_CON, uri_path_options,
				payload, payload_size, &request, buf);
	if (ret < 0) {
		LOG_ERR ("CoAP init failed: %d", errno); // <---------------- ERROR LOG
		goto end;
	}

	if (reply_cb != NULL) {
		coap_set_response_callback(&request, reply_cb);
	}

	ret = coap_send_message(addr, &request);
	if (ret < 0) {
		LOG_ERR("Transmission failed: %d", errno);  // <---------------- ERROR LOG
		goto end;
	}

end:
	return ret;
}

I was referring to the error logs as marked above. I dont see any of these errors in my case. So I assume that 'coap_send_request ()' executes without any error. Am I correct?

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/480697"]That would be if send_sensor_update() is not called. Do you have something indicating whether or not these are called at the time when the devices become unavailable?[/quote]

I dont this this is the case as I can see this log message continuously from a disconnected SED.

LOG_INF ("ZS %d, RSSI %d, LQI %d, LQO %d, FW %04x", the_sensor_device->zoneState, RSSI, linkQalIn, linkQualOut, FWRevNum);

So it seems like my application code gets called continuously but data is not being send from that point onwards.

When we look at the console of the host, we couldn't see the log message for the data receive from these disconnected sensors. The disconnection could happen from

1. sensor thread stack

2. host thread stack

3. host application

Do you see any other ways?

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

Edvin — Thu, 25 Apr 2024 12:58:21 GMT

[quote user="kaushalyasat"]When I look into 'coap_send_request (...)', it has 'coap_init_request (...)' and 'coap_send_message(...)' functions. Both these has error logs printed if something goes wrong[/quote]

I was thinking about these. What error logs do you refer to?

[quote user="kaushalyasat"]Can you think of any ways that coap_send_request (..) may not have been called?[/quote]

That would be if send_sensor_update() is not called. Do you have something indicating whether or not these are called at the time when the devices become unavailable?

I am sorry, but we are several months into this, and I am not quite sure what we are discussing anymore. You have some devices in a remote area that you do not have physical access to where you see some devices drop out from time to time, right? Perhaps you can try to replicate this in a local area where you have access to your devices? Reset the entire network, start sniffing before you start your devices so that the sniffer can capture everything from the beginning. Then it should be able to pick up and resolve all the short addresses. When you detect the issue, look into the log from that particular device. Does it say anything when trying to call coap_send_request()? Any error messages? coap_send_request() also returns a value based on how it did. It returns 0 on success, and a negative number on failure. Try printing something in the log in the cases where this returns < 0. What does it return when it fails?

Best regards,

Edvin

RE: Mesh back and forth seems to break connection

kaushalyasat — Thu, 18 Apr 2024 23:42:35 GMT

Hi Edvin,

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/479544"]I don't think it is possible to rearrange the packets like this, no.[/quote]

As I mentioned, in this thread, one of Nordic engineers mentioned something like that - nRF Sniffer integration for 802.15.4 in a python scipt (Pcap file problems)

Also I can see in wireshark, you can time shift packets around like this.

I tried doing this, but couldn't see any effect. I am not sure my operation was correct, so I leave this for a wireshark guru to comment.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/479544"]What error messages did you see?[/quote]

What you mean is error messages I saw in the RTT viewer? I didn't see any.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/479544"]Are they printed from within the implementation of coap_send_message()?[/quote]

Yes. If you look at coap_send_request (...) function in coap_utils.c, you can see that coap_init_request (...) is logged from within and coap_send_message (...) return is logged. I didn't go any deeper as I got stuck in z_impl_zsock_sendto (...) in sockets.c.

[quote userid="26071" url="~/f/nordic-q-a/107621/mesh-back-and-forth-seems-to-break-connection/479544"]You also need to consider the possibility that for some reason these functions aren't called at all, due to some other error.[/quote]

static void send_sensor_update (struct k_work *item) {
    .
    .
    .
    
	LOG_INF ("ZS %d, RSSI %d, LQI %d, LQO %d, FW %04x", the_sensor_device->zoneState, RSSI, linkQalIn, linkQualOut, FWRevNum);

	memcpy (&payload[1], myExtAddr.m8, 8);
	memcpy (&payload[9], myEUI64.m8, 8);

	payload[17] = ((the_sensor_device->temp)>>8) & 0xff;
	payload[18] = (the_sensor_device->temp) & 0x00ff;

	payload[19] = ((the_sensor_device->vbat)>>8) & 0xff;
	payload[20] = (the_sensor_device->vbat) & 0x00ff;
	payload[21] = RSSI;
	payload[22] = linkQalIn;
	payload[23] = linkQualOut;
	payload[24] = the_sensor_device->zoneState;
	payload[25] = (uint8_t)(FWRevNum >> 8);
	payload[26] = (uint8_t)(FWRevNum & 0x00ff);

	ARG_UNUSED(item);

	if (net_ipv6_is_addr_unspecified (&unique_local_addr.sin6_addr)) {
		LOG_WRN("Peer address not set. Activate 'provisioning' option on the server side");
		return;
	}

	coap_send_request(COAP_METHOD_PUT, (const struct sockaddr *)&unique_local_addr, sensor_option, payload, sizeof(payload), NULL);

This is the code section from the log print to coap_send_request (...) in my code. Can you think of any ways that coap_send_request (..) may not have been called? if IP6 is missing, I would get a LOG_WRN, which I dont get.

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

Edvin — Thu, 18 Apr 2024 20:21:48 GMT

Hello Kaushalya,

I don't think it is possible to rearrange the packets like this, no. If anything, I think you woudl have to edit the raw .pcapng file.

[quote user="kaushalyasat"]

Also as I mentioned earlier, we dont see any other error messages in my RTT viewer. When I look into 'coap_send_request (...)', it has 'coap_init_request (...)' and 'coap_send_message(...)' functions. Both these has error logs printed if something goes wrong. So can we assume that there were no errors reported during a packet transmission?

[/quote]

That depends on the implementation. What error messages did you see? Are they printed from within the implementation of coap_send_message()? Or from the function that checked their return value?

You also need to consider the possibility that for some reason these functions aren't called at all, due to some other error.

Best regards,

Edvin

RE: Mesh back and forth seems to break connection

kaushalyasat — Thu, 18 Apr 2024 01:13:49 GMT

Hi Edvin,

I am building another FW with return values gets logged. Unfortunately we have to wait until this problem creeps up again since we dont know how to recreate it.

Is there a way to rearrange wirechark packets so that we can push a MLE packet up? Then the subsequent logs can get extended address from that.

how to reset the network? What APIs should I use?

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

Edvin — Wed, 17 Apr 2024 07:20:21 GMT

[quote user="kaushalyasat"]Can you shed some light?[/quote]

I believe the takeaway from this thread is that if the sniffer didn't pick up the packets where the nodes joined the network (for the first time), so that it uses it's extended address and was assigned a short address, the sniffer doesn't know how to map the short RLOC16 addresses to the extended addresses, and hence, it can't decrypt the packets to/from these devices.

[quote user="kaushalyasat"]Here I have not handled the return value from coap_send_request (),[/quote]

So is it possible to check these return values? If you don't see the packets, it may mean that they are never sent. And if that is the case, then a clue is probably found in the return value from this function.

[quote user="kaushalyasat"]How can we further debug this?[/quote]

Check the return value for coap_send_request() when the packets aren't sent correctly.

Then, try to reset the entire network, or at least factory reset the sensor devices, so that they do a new provisioning sequence when you turn them on. Then you need to enable the sniffer before you provision the devices, so that the sniffer can pick up the extended addresses being used before they are assigned an RLOC16 (short) address. You can experiment with this in a small scale in your office. Set up a small network with two devices, and try starting the sniffer before the provisioning process, and after, and compare the results.

Best regards,

Edvin

RE: Mesh back and forth seems to break connection

kaushalyasat — Wed, 17 Apr 2024 04:36:45 GMT

Also from the log I think I can see that the sensors (SEDs) which has disappeared from the network actually is connected to the network from the sensors point of view. I can see my log message just before calling the coap_send_request (). Following is the code section.

...
LOG_INF ("ZS %d, RSSI %d, LQI %d, LQO %d, FW %04x", the_sensor_device->zoneState, RSSI, linkQalIn, linkQualOut, FWRevNum);
...
coap_send_request(COAP_METHOD_PUT, (const struct sockaddr *)&unique_local_addr, sensor_option, payload, sizeof(payload), NULL);
...

Here I have not handled the return value from coap_send_request (), which is my bad. But I dont get any error logs from this. But I cant see the packet being transmitted in my wireshark logs. So I have the feeling that this is something related to either CoAP stack or the OpenThread stack.

There could be possibility that I dont see the actual packet in wireshark log because I dont have sufficient data to filter these. As I said, I dont know the RLOC of these sensors, I only know their MAC. But I cant easily filter based on the MAC as I cant see it in any frames.

How can we further debug this?

Cheers,

Kaushalya

RE: Mesh back and forth seems to break connection

kaushalyasat — Wed, 17 Apr 2024 04:27:12 GMT

Also I came across this thread from an old devzone ticket.

nRF Sniffer integration for 802.15.4 in a python scipt (Pcap file problems)

Here Nordic engineer mentions that extended address is required in a packet to decrypt and this can be 'fixed' by moving packets with extended address to the top. I tried doing this with the attached log, but I am not 100% sure how to do it.

Can you shed some light?

Thanks,

Kaushalya