Multiple nodes acknowledgement problem

RE: Multiple nodes acknowledgement problem

Hung Bui — Wed, 03 Jun 2020 13:19:20 GMT

Hi Dinesh,

What is count.occur ?
Could you explain a little bit more, for example what would average data missing count of 165 meant ? Maybe you can use for example 5/25 for example, meaning 5 failed out of 25 trials ?

A flooding Mesh network usually rely on redundancy to keep the network reliable. Meaning there should be overlapping of the coverage of multiple nodes. The more nodes you have the more transmission and more chance that the packet will not be lost.

In your case you are testing with only 2 (or 3 devices) with not much over lapping coverage. There will be packet drop if there is any interference.
So my suggestion here is:

- Increase the number of nodes.

- Increase the retransmission rate. In this case you can increase CORE_TX_REPEAT_RELAY_DEFAULT inside nrf_mesh_config_core.h. By default it's 1 meaning only one transmission as a relay (or proxy node). You can try increase it to 4 to see if the packet drop rate is lower ( i tested here and it's lower).
But the draw back will be more data traffic in the mesh network. You should only do this when there isn't many node in the same area, otherwise it will cause congestion on the network.

- In addition you can increase CORE_TX_REPEAT_ORIGINATOR_DEFAULT as well, it will increase the number of retransmission of the packet originally from the node.

- Turn off Proxy role on the node that's not expected to connect to the phone. Not all the nodes needed to connect to the phone, you can turn off Proxy role on these nodes. MESH_FEATURE_GATT_PROXY_ENABLED = 0.

- Increase TX_POWER as suggested in the last reply.

Could you tell a little more about your application ? How many nodes, how large is the space ? How should the phone connected to the network and control the devices ?

RE: Multiple nodes acknowledgement problem

Dinesh — Wed, 03 Jun 2020 06:22:31 GMT

Hi Hung,

Please note that we have taken 25 trials and calculated it's average count.occur

Also can you please let us know, how you have solved timeout issue which occurred at your side (as mentioned by you in below quoted message)?

[quote userid="2121" url="~/f/nordic-q-a/58969/multiple-nodes-acknowledgement-problem/250611"]

Hi Anaya,

I just tried here with the stock example and found the same issue. Most likely it's the update on the app caused the issue. I will continue the test and let you know if we find the root cause.

[/quote]

Regards,

Dinesh

RE: Multiple nodes acknowledgement problem

Hung Bui — Tue, 02 Jun 2020 08:25:58 GMT

Hi Dinesh,
Thanks for the information. Could you give some more detail about how you calculate the "Average Data Missing Count" , how many trials did you perform to get the result ?

RE: Multiple nodes acknowledgement problem

Dinesh — Sat, 30 May 2020 12:40:12 GMT

Hi Hung,

Greetings!

Please note my reply to your queries as below.

Before that I wish to clear one point related to BLE module is that. We have two types of BLE modules having NRF52832. So we have conducted trials for both modules and observed the result as follows.

test #1 is done (both indoor and our door, close range and long range) You can start testing with your hardware, and only use unmodified light switch server firmware for testing.

Herewith please find following table for data missing observations. Average data missing count indicates that after specified operations data will miss.

Sr.No.	Node Setup Description	Application	Range	Average Data Missing Count
1	Bluetooth controller with Old module	Gladiance	Short	12
			Long	2
2	Development Board	Gladiance	Short	32
			Long	12
3	Bluetooth controller with New module	Gladiance	Short	165
			Long	15
4	Development Board	Stock App.	Short	72
			Long	60

Bluetooth controller with New module, Gladiance Application for short range is working very well. Where as Bluetooth controller with Old module with Gladiance Application for long range is working very poor.

test #2, you can try to increase the TX_POWER of the node, if the power consumption is not a big issue. To change TX_Power, you can modify set_default_broadcast_configuration() to set it to RADIO_POWER_NRF_POS4DBM instead of RADIO_POWER_NRF_0DBM.

We'll get back to you on this shortly after concluding on test #1 as we wish to have some safe margin for increasing range.

Regards,

Dinesh

RE: Multiple nodes acknowledgement problem

Hung Bui — Mon, 25 May 2020 09:01:06 GMT

Hi Dinesh,

It's difficult for us to look at the log and figure out what exactly happened. The reason is that we don't know at which period the time out happened.

But from what you described I have some thoughts as follow:

- There is a chance that the proxy is not in listening mode all the time as it supposed to do.

- But the issue could also be the radio hardware you have on the board, as you described the range you are testing is only 15-30 feet and I assume it's the max range you can achieve, then it's too short. With a tuned antenna it can achieve at least 50feet.

- To make sure it's the ACK that didn't receive by the proxy, please clarify that when you see the timeout issue occurred on the phone, you do see that the destination node actually turn off/on as the command from the phone. It's just the phone didn't receive confirmation , but the actual command is executed (the light turn on/off as expected).

The case has been dragging for quite some time. I would strongly suggest you to do the following test:

- Test using only nRF52 DK. I can see that you have at least 2 nRF52 DK. One as the server and one as the sniffer. Please use both of them as server for testing. And please use unmodified light switch server for testing. This test is very important.

- After test #1 is done (both indoor and our door, close range and long range) You can start testing with your hardware, and only use unmodified light switch server firmware for testing.

- After test #2, you can try to increase the TX_POWER of the node, if the power consumption is not a big issue. To change TX_Power, you can modify set_default_broadcast_configuration() to set it to RADIO_POWER_NRF_POS4DBM instead of RADIO_POWER_NRF_0DBM. This would increase the signal power and overcome the interference. But if it's the issue that the proxy doesn't listen, then we should see no change in performance.

RE: Multiple nodes acknowledgement problem

Dinesh — Sat, 23 May 2020 11:44:13 GMT

Hi Hung,

Greetings!

Please note my reply to your queries as below

- Could you give more information about the range you achieved in the test ?
For indoor trials: Distance between client device and server devices are about 15 feet and distance between to servers is around 15 feet.

- What was the range achieved in the test outdoor ? Especially the test with 3 devices.
For outdoor trials: Distance between client device and server devices are about 50 feet and distance between to servers is around 15 feet.

- Have you tried to test the same with only the nRF52 DK (on both 2 servers) ? The reason for this is to avoid any RF issue on your hardware board.
No we have not conducted trials using nRF52 DK.
So considering this and wireshark filetrs as suggested by you we have taken trials on nRF52 DK.
Details for this setup is as below

Devices:
a. Client Device: Nordic nRF Mesh Android App
b. Server Device: nRF52 DK based modified Light Switch Server (2 nos), distance between this 2 devices is 30 feet
c. Sniffer Device: nRF52 DK based

Setup:
a. As suggested by you, we have kept sniffer near to proxy device to which android app is connected
b. 2 Servers are kept at different locations with distance between them is 30 feet.

Observations:
a. Then we have captured wireshark sniffer log and device RTT log for proxy node.
b. We tried to read and understand wireshark capture and rtt log after timeout issue occurred while operating from android app.
c. If we have read these logs correctly then it is something like below
app send command to proxy node --> proxy node transmitted on mesh network --> proxy node did not receive ACK command
d. Here we do not know whether end node has received command and responded to that command,
so in another trial we tried to catch end node device rtt log.
(sorry to day but it was not possible for us (rtt) log all devices at a time, so we have taken end node rtt log in separate trial).
e. Here we have found that end node has transmitted ACK command to APP

Conclusion:
a. Timeout issue: As per our analysis it looks like that end node transmits ACK but it do not get received by proxy node.
b. Data missing rate: On nRF52 DK based setup, it is observed that data missing rate is 1 in 50 operations and that too without re-transmit logic at android app side.

Request you to please go through attached log file and guide us on the same.
I hope these trials are sufficient for pointing out the possible issue and to provide fix.

Attachements:
TRIAL_1 to TRIAL_3: Wireshark captures and proxy node rtt log
TRIAL_4: End node device rtt log

Regards,

Dinesh

devzone.nordicsemi.com/.../20200523_5F00_151300.zip

RE: Multiple nodes acknowledgement problem

Hung Bui — Fri, 22 May 2020 15:04:04 GMT

Hi Dinesh,

Thanks for the intensive test information.

Could you give more information about the range you achieved in the test ?

- What was the range achieved in the test outdoor ? Especially the test with 3 devices.

- Have you tried to test the same with only the nRF52 DK (on both 2 servers) ? The reason for this is to avoid any RF issue on your hardware board.

The success rate you achieve is much lower that what we observed here.

Another test you can do is to use the sniffer to track the Mesh packet. If you use this filter: btcommon.eir_ad.entry.type == 0x2a like this:

You should be able to filter out only ADV mesh packets.

You can see a successful transaction (turn on or off the light) should be something like this:

In this device 86:06 is the one who connected to the phone, and the 29:17 was the destination node.

If it failed either the initial packet from 86:06 would be missing, or the 86:06 is received by the sniffer but couldn't receive by the 29:17.

By doing the trace we can observe if the problem happen because the proxy node doesn't send packet, or it's the packet was not received by the destination node or both. You can try to leave the sniffer very close to the proxy node.

RE: Multiple nodes acknowledgement problem

Dinesh — Fri, 22 May 2020 11:59:19 GMT

Hi Hung,

Greetings!

As you suggested we have taken trials with below setup
SDK: v4.0.0
Client: Stock SDK Light switch application on nRF52DK hardware
Server: Modified SDK Light switch server application on custom hardware/pcb
Mobile App: nRF Mesh App on Android

Trials are as below

1. Indoor trials
a. From Embedded switch client (Client device)
i. 1 Server and 1 client in setup (Client Device -------> 1st Server)
It is observed that there is no data miss with this device.

ii. 2 Server devices and 1 client device (Client Device -------> 1st Server -------> 2nd Server)
Here data miss or timeout issue occurred (data missing rate is around 20%)
We have placed devices in such a way that 2nd server is not operable from client device.
In this case we have observed data miss issue (operated 2nd server) and as per log it is observed that
ACK from end device is received at 1st server but it is not forwarded to client device,

b. From Mobile App
i. 1 Server and 1 client in setup (Client Device -------> 1st Server)
No data loss occurrs as client device (mobile app) is directly connected to end node

ii. 2 Server devices and 1 client device (Client Device -------> 1st Server -------> 2nd Server)
Data missing issue occurred (data missing rate is above 50%).
As per log it is observed that at some time proxy node forwarded command but end node did not received it,
and at some time end node send back ACK command but proxy node did not received it.

2. Outdoor trials
a. From Embedded switch client (Client device)
i. 1 Server and 1 client in setup (Client Device -------> 1st Server)
ii. 2 Server devices and 1 client device (Client Device -------> 1st Server -------> 2nd Server)
It is observed that there is no data miss issue if devices are kept at line of sight.
However if there is change of 2-3 feets change in line-of-sight resulted in 15-20% data missing.

Request you to please provide us possible resolution to this issue.
We are waiting for your positive reply on the same.
Thank you once again for your extended help to us.

Regards,

Dinesh

RE: Multiple nodes acknowledgement problem

Hung Bui — Wed, 20 May 2020 12:19:23 GMT

Hi Dinesh,

It's not possible for us to debug based on the log screenshot. I would suggest you to check and confirm that the proxy node sent the packet but the destination node couldn't receive the packet.

Then the next step is to test normal mesh communication (instead of sending command from the phone, you send the mesh command directly from one node to another node, and check the failure rate of this method).

My suggestion on testing on different environment is to do the same test at a location with less Wifi (or other RF) traffic. Check for other bluetooth device in the building, if there is a device that keep transmitting all the time at low interval it can cause the problem. So it's important to test on a clean enviroment (try to test outside, in an open area for example)

Please try to test using more devices in the network. This will improve the redundancy and the packet drop rate will be lower.

RE: Multiple nodes acknowledgement problem

Dinesh — Wed, 20 May 2020 10:15:49 GMT

Hi Hung,

Greetings!

Please note that we have taken trials as suggested by you (with SDK4.0.0).

1. We have taken trials keeping Wi-Fi network On and Off
2. We have enabled log as you told
3. We are also experiencing that proxy node forwards command to end node but at end node it looks like packet is not received
4. Also we have observed that decrypt status for network RX packet is '5' (NRF_ERROR_NOT_FOUND). You can see this log in screenshot as "Net RX decrypt status: 5"

Please see attached screenshot(s) for your reference.

Regards,

Dinesh

RE: Multiple nodes acknowledgement problem

Dinesh — Wed, 20 May 2020 04:37:12 GMT

Hi Hung,

Greetings!

I am colleague of Mrs. Anaya, working on nRF52 part.

As suggested by you, we have to debug this issue in more better way so let me introduce myself in this communication.

Please see my reply as below considering your above queries.

1. We have Wi-Fi network in our company with 2.4GHz frequency (auto mode channel).

2. Also please note that yesterday we had trials and have also experienced same thing which you had. At first trial we had good communication response but in next trial it was worst experience.

Here let me understand that if Wi-Fi network is causing for data loss then what can we do to improve this?

3. In our test setup we are using 2 devices under test. However in our lab there are more Bluetooth devices are active say about 8-10 devices which are being used by my colleagues (provisioned with different android devices)

Here let me understand below point as you asked for number of devices.

Is increase in number of devices lead to decrease communication response/performance?

4. We have also test it on nRF52 DK and have also experienced this timeout error

5. We will take trials with logging enabled as you suggested and will let you know the test results.

6. At present we are using SDK 4.0.0 with modified light switch server application.

I think it would be for us better to test using this application as we had more trials with this setup till now.

7. Can you please explain bit more on what do you mean by different environment?

So that it will help us to take trials precisely.

8. One more point which I would like to share with you that. Yesterday we have trials with devices which do not go for flash write operations (to save device status).

Here we have seen improvement in communication but it is not that worth to say that problem is resolved. Here improvement is about 10% only

Thank you so much.

Regards,

Dinesh

RE: Multiple nodes acknowledgement problem

Hung Bui — Tue, 19 May 2020 15:00:07 GMT

Hi Anaya,

I have continued the test but the result was not consistent. So at a point I have 60% packet dropped but when I tested again now it worked like before about 5-10% drop. I suspecting the interference from many of our test devices in the building caused the issue.
I would suggest you to check that as well. Do you have any high traffic on the 2.4GHz domain ? like wifi and other network ?

In your test how many devices do you have ? Have you tried to test using NRF52 DK ?

We need to have a better way to debug this issue. I think it's the best to print out logging.

Please init logging like this in the light switch application:

__LOG_INIT(LOG_SRC_APP | LOG_SRC_FRIEND|LOG_SRC_BEARER|LOG_SRC_NETWORK, LOG_LEVEL_DBG1, LOG_CALLBACK_DEFAULT);

We will stick with the light switch application for testing to make it easier to align the test on your side and our side. If you can test using a fresh copy of the SDK it would be nice. I assume you are testing on SDK v4.0 ? Please also consider testing on v4.1 as well.

In the RTT log you will be able to find logging from proxy.c and from network.c.
The proxy.c handle the connection to the phone. And network.c is the actual mesh ADV packet between nodes.

In my testing here, I can see that when there is a command from the phone (that got timeout) the proxy node has managed to forward it to the network layer and the network layer sent the packet.

But on the remote node (not the proxy node) I don't see it receives this TX packet, showing that the packet is dropped/corrupted, could be due to interference.

Please try to test with debug enabled and let me know what you find. Also please try to test on different environment, and with different phone, etc.

RE: Multiple nodes acknowledgement problem

Hung Bui — Tue, 19 May 2020 08:34:18 GMT

Hi Anaya,

I just tried here with the stock example and found the same issue. Most likely it's the update on the app caused the issue. I will continue the test and let you know if we find the root cause.

RE: Multiple nodes acknowledgement problem

Anaya — Mon, 18 May 2020 12:09:29 GMT

Hi Hung,

Greetings!

Thanks for your reply and suggestions.

As per your suggestion we have taken trials with modified and stock applications (light switch server) and captured sniffer log for device to which android app is connected. (2 trials for each modified and stock application). Hope now we have given you the correct log files.

Please note that we have experienced timeout log in both the situations.

Request you to please suggest us further steps to resolve this issue.

Regards,

Anaya

devzone.nordicsemi.com/.../Wireshark_5F00_capture_5F00_files_5F00_20200518_5F00_165500.zip

RE: Multiple nodes acknowledgement problem

Hung Bui — Mon, 18 May 2020 09:12:50 GMT

Hi Anaya,

The sniffer trace you provided didn't track any connection.

You need to select the advertiser before the connection. Inside the Interface menu, there is a dropdown list of all advertiser, you need to choose the one you want to sniff. After that you make the connection from the phone. Please follow the instruction in the sniffer manual. Most important thing is that you can see the connection established in the sniffer trace. In your current traces there are only advertising packet.

Do you see the same issue when you test with our stock light switch example (no modification) ?

RE: Multiple nodes acknowledgement problem

Anaya — Sat, 16 May 2020 09:59:09 GMT

Hi Hung,

Greetings!

1. As per your suggestion we have tried sniffer trace and have taken few trials with 3 different devices.
We have connected only 2 devices at a time and operated 2nd device by connecting app to first device.
Out of these 3 devices 1 is nRF52DK and rest 2 are our devices.

Also we have used nRF Mesh Android App for these trials.

Please find attached Wireshark Capture Log files for your reference and guide us further.

2. Also please note that we are not using any long interrupt handler in our device firmwares

Regards,

Anaya

devzone.nordicsemi.com/.../Wireshark_5F00_capture_5F00_files.zip

RE: Multiple nodes acknowledgement problem

Hung Bui — Fri, 15 May 2020 09:00:19 GMT

Hi Anaya,

Here is the reply from your colleague Madhav that I quote:

We are seeing this issue once in a 10 times in Android application.

Also we have taken trials with iOS application where in we have operated for approximately 200 times and still there was no single occurrence of timeout error.

So what have changed ? It seems to be much worst now.

What you can do is to test with our light switch example and check if the proxy node has the same issue. Also please try to test using our NRF52 DK just to align the hardware with what we can test here.

What you described is much worst than what we are observing here, we also see the issue only once in about 10-20 trial, not 6-8 times out of 10. There must be something else that affected this.

A possibility is that the connection interval on BLE connection to the phone was too small. If it's too small it will occupies the time domain reserved for mesh. Could you try capture a sniffer trace of the connection ?
Or do you have any long interrupt handler that can cause any affect on the mesh performance ?

RE: Multiple nodes acknowledgement problem

Anaya — Fri, 15 May 2020 06:14:35 GMT

Hello,

For the point 2,

When you test with the iOS phone do you see the same issue that it takes much longer time to send a command to remote node compare to the proxy node ?

We again checked in nRF mesh app for iOS, we found the same problem. It doesn't show the timeout error, but the lamp on/off not works, its missing packets for the other nodes in the network except 1st node to which it's connected.
For the 1st connected node it works perfectly without the packet loss.

Also we the same issue in android and iOS application that, for the other nodes in the network the problem is same for configuration messages such as binding app key, publication address etc.

Please give the appropriate solution as soon as possible.

Regards!

RE: Multiple nodes acknowledgement problem

Anaya — Fri, 15 May 2020 04:05:42 GMT

Hi,

1. Let's test without the retry mechanism, how often do you see the issue ? When the issue happened did you see the destination node turn off/on the light (meaning only the ack message was missing) ?

-> without retry mechanism sometimes, on/off server switch on/off the light and not gives acknowledgement, this case occurrs approximately 3-4 times out of 10.
And say about 6-7 times out of 10, it's on/off the light but does not give acknowledgement

2. When you test with the iOS phone do you see the same issue that it takes much longer time to send a command to remote node compare to the proxy node ?

Need Ito check will tell about this after sometime.

3. We need also to test the mesh operation without using the phone/proxy, please try to test using our light switch example, where you press a button on the client it can control the server. This way we can rule out any issue with the phone or the proxy protocol.

Checked, it works perfectly without with light switch client without phone.

What I'm suspecting here is that the mesh traffic has lots of packet drop, either because of the hardware or because of the interference from other network (wifi, etc) we need to make it clear that the normal mesh operation can work fine before testing with the phone.

I agree with you. It works fine without phone.

Thank you.

Regards!

RE: Multiple nodes acknowledgement problem

Hung Bui — Thu, 14 May 2020 09:27:27 GMT

Hi Anaya,

We need to clarify this.

Let's test without the retry mechanism, how often do you see the issue ? When the issue happened did you see the destination node turn off/on the light (meaning only the ack message was missing) ?

When you test with the iOS phone do you see the same issue that it takes much longer time to send a command to remote node compare to the proxy node ?

We need also to test the mesh operation without using the phone/proxy, please try to test using our light switch example, where you press a button on the client it can control the server. This way we can rule out any issue with the phone or the proxy protocol.

What I'm suspecting here is that the mesh traffic has lots of packet drop, either because of the hardware or because of the interference from other network (wifi, etc) we need to make it clear that the normal mesh operation can work fine before testing with the phone.

RE: Multiple nodes acknowledgement problem

Anaya — Thu, 14 May 2020 08:46:17 GMT

Hello,

Yes as you suggested,It works after retries, but it is very slow. Most of the times it on/off the lamp after 2nd or 3rd reply and in 1 out of 30 operations not executed at all after 3 reply also, where as the 1st node on which it connects 1st, it works very fast i.e not required to retry.

So that the for other nodes operations it gives very bad user experience.

My requirement is that the operating experience should be same for all nodes in the network.

Then why it is different for 1st and other nodes as they all are configured in same mesh network.

Is it required to connect to each node separately?

Please guide and please give the solution as early as possible.

Thank you!

Regards!

RE: Multiple nodes acknowledgement problem

Hung Bui — Wed, 13 May 2020 12:30:48 GMT

What exactly didn't work ? please describe.
You mentioned before the issue only happens once in 10 times, so with a few retries, it should only happen once in 1000 times or more.

RE: Multiple nodes acknowledgement problem

Anaya — Wed, 13 May 2020 11:15:27 GMT

Hello,

As per your suggestion I tried the retry logic, but it didn't work as expected.

Please suggest some solution.

Thank you.

Regards!

RE: Multiple nodes acknowledgement problem

Hung Bui — Mon, 20 Apr 2020 14:46:29 GMT

Please modify the Android app to send the retry logic. So instead of having a long timeout and wait for the ACK message. You should have shorter timeout (1/3 of the original) and when it timed out, you send a new message. And after 3 times you then can throw the actual timeout to the GUI interface.

RE: Multiple nodes acknowledgement problem

Anaya — Mon, 20 Apr 2020 13:46:42 GMT

Hello Sir,

I have checked in iOS app, the retry command goes from the NRF mesh library itself, but in android application it is not implemented.

I want to solve this issue in android application, then where we should implement retry logic, in NRF mesh library for android or in example app?

Please guide us.