Need help to improve Bluetooth Mesh throughput

RE: Need help to improve Bluetooth Mesh throughput

David Edwin — Tue, 22 May 2018 10:12:30 GMT

Just to summarize, you are looking at a

1. Throughput of about 200 kbits per 10 second per node (25 KBytes per 10 seconds per node)

- This will mean that the bandwidth at the gateway/exit node from the mesh is number of nodes multiplied by the per node bandwidth , i.e. for 15 nodes it would mean 15 x 200 kbits per 10 seconds , i.e. 3000 kbits per 10 seconds or 300 kbits per second.

2. Number of nodes sending data : assumed to be 15

- We assume 1 exit node in addition

3. Maximum number of hops : Let me assume 2 hops

As you can see the throughput at the gateway node will need to scale for the number of nodes, so you need to look at that to see if it is viable for your products final configuration or examine ways to reduce data being generated from the nodes.

More hops will also degrade the bandwidth substantially even with instaburst

To test the instaburst, I would suggest using only 2 nodes first, i.e. 1 node sending the data and 1 node receiving the data as gateway or exit node measure the time taken and then add more nodes to understand the behavior that you see. Additionally get the statistical data from the mesh so you see where the bottlenecks are.

Your usecase is using significant bandwidth and I think will need tuning and good understanding of hops and node count to get the numbers you want, so approach this carefully.

You can also consider using the Fruity Mesh as the mesh backend for your project as that is connection oriented and could get better throughput.

RE: Need help to improve Bluetooth Mesh throughput

vishal — Mon, 07 May 2018 12:45:26 GMT

Hi aleccontrol,

I have facing same issue in my current project i have developed complete application using nRF mesh technology. But my sensor data is 24KB this all data is transmitted but it took near about 2 minutes. But this time is much more we want to send this all data within 5-6 second. Can you tell me is it possible in nRF mesh technology. currently i have using mesh experimental instabrust feature for increase throughput and larger bandwidth. using this feature it took 2 minutes.

Can anyone tell me is it possible to transmit 24-25 KB data within less than 10 second in mesh network with relaying functionality.

Thanks..

RE: Need help to improve Bluetooth Mesh throughput

mike — Tue, 01 May 2018 09:04:31 GMT

Is there any need to change something in the node configuration or node setup on the provisioner side?

RE: Need help to improve Bluetooth Mesh throughput

leonwj — Mon, 30 Apr 2018 11:43:16 GMT

Hello,

Yes, from recollection the code was built on top of the light switch example in v1.0.1 of the mesh SDK. I recall that a few changes were made to get it up and running but the custom model was left intact.

Regards,

RE: Need help to improve Bluetooth Mesh throughput

mike — Mon, 30 Apr 2018 07:54:47 GMT

Hello, your'e saying you had no issues with the attached code, so naturally I'm wondering where the difference lies=) do you call send_command_client() from main.c or how have you set it up?(newbie here). I suppose you built it on top of the light switch example?

RE: Need help to improve Bluetooth Mesh throughput

David Edwin — Mon, 30 Apr 2018 07:51:50 GMT

Have you got some data on the packet losses from the statistical interface so you can be sure if it is an RF side packet loss or an internal buffer restriction.

RE: Need help to improve Bluetooth Mesh throughput

David Edwin — Mon, 30 Apr 2018 07:51:48 GMT

Have you got some data on the packet losses from the statistical interface so you can be sure if it is an RF side packet loss or an internal buffer restriction.

RE: Need help to improve Bluetooth Mesh throughput

David Edwin — Mon, 30 Apr 2018 07:51:48 GMT

Have you got some data on the packet losses from the statistical interface so you can be sure if it is an RF side packet loss or an internal buffer restriction.

RE: Need help to improve Bluetooth Mesh throughput

aleccontrol — Mon, 30 Apr 2018 00:24:11 GMT

I have not been able to get any further improvement in the performance. It appears to me that either the BT Mesh or the Nordic stack is not be able to handle the throughput I need for my application (which really is nothing more than to replace serial connection at a modest at 9600bps). The large latency and high percentage of packet loss I experienced makes it unsuitable for a command/response communication exchange between two device via the BT mesh. I will probably have to look for another technology.

RE: Need help to improve Bluetooth Mesh throughput

mike — Sun, 29 Apr 2018 18:23:42 GMT

Hello

I am sitting with the exact same problem, with a similar setup as you. Did you get any further on your original issue? Is it only packet loss that is the problem or did you find anything else in code?

RE: Need help to improve Bluetooth Mesh throughput

Hadi Deknache — Wed, 25 Apr 2018 16:06:24 GMT

Hi, this might be an off topic question, but we were testing this and wanted to log something similar to what is done above with timestamps when starting and stopping. How did you achieve this when running Segger J-Link RTT Viewer?

Thanks in advance :)

RE: Need help to improve Bluetooth Mesh throughput

aleccontrol — Sat, 24 Mar 2018 07:11:53 GMT

Hi Leonwj,

I appreciate that you took time to reply and provided some useful statistics. I agree that your test case was pretty intense since you are using 3 servers to trigger the client to send command every 100ms so the air wave is probably pretty congested.

In our case we are just sending the command out of the client (the frequency of sending the command is entirely controlled by the client) and then the client will wait to receive the response back from one single server. Only when the client has received a response from the server will it continue to send another command. This is pretty common method in industrial control where the master (client) send a command to a slave (server) and then wait for the slave to return the response before continuing further.

Would you be able to test such a setup and determine the latency that is expected for a complete command/response exchange? We are interested in knowing how many command and response exchanges that can realistically be accomplished via the BLE mesh to determine if we could use BLE Mesh as a communication media.

What we experienced is that it seems impossible for the client to sustain a continuous command/response exchanges with the server without frequent errors. One in every few commands that were sent out by the client did not get a response from the server and so the client side blocks until time-out (set at 2 seconds). In our test case we are sending just a few bytes (below 11 bytes threshold before SAR kicks in) and in real life the client and server will be sending and receiving many more bytes per exchange so it is important that latency and packet loss are kept to minimum.

In our application the client may occasionally need to retrieve say 2000 bytes of data from a server. What would be best the way to achieve it in the shortest possible amount of time? In traditional communication via UART the client will send a command packet that tell the server to return say up to 100 bytes per response packet and so it will take about 20 successful exchanges to retrieve 2000 bytes. How long would you expect it will take to retrieve this amount of data?

Thank you very much!

RE: Need help to improve Bluetooth Mesh throughput

leonwj — Sat, 24 Mar 2018 01:05:19 GMT

Additionally, (for both yourself and anyone else who's interested in the throughput level expectations of Bluetooth Mesh), you might want to listen to the following podcast/webinar in which Szymon Slupik, Chair of the Bluetooth SIG Mesh Working Group highlights why he believes that Bluetooth Mesh meets the required performance levels (among other benefits) for both industrial and home.

(please also note that I'm not affiliated with Szymon and/or Mr. Beacon in any way but have simply followed the mesh specification process since FRD 0.7)

Regards,

RE: Need help to improve Bluetooth Mesh throughput

leonwj — Sat, 24 Mar 2018 00:41:24 GMT

hello aleccontrol,

Apologies for the belated response to your latest comment...

Certain things that I wanted to point out regarding the packet loss statistics that I presented, are as follows:

1. the stats were from using the model that you uploaded without any changes, so we would possibly need to review the model before attributing any loss to the underlying mesh

2. We implemented the cmd/resp mechanism under the heartbeat callback mechanism, which may not be the most appropriate place to do so. e.g. if we set the heartbeat period to 100 msecs for each server, that means that when each server sends a message, the client immediately publishes a "Test" command to the group (i.e. 3 servers) which then each send back a response. So every 1/10th sec we have:

a. 3 heartbeat messages (server(s) -> client)
b. 1 command message published to the group (client -> server(s))
c. 3 response messages sent to the client (server(s) -> client)

Given that there shouldn't be any relaying taking place (the nodes are within close proximity and message caching would ensure that repeats are avoided), that means 70 messages every second across the mesh. Also, we haven't done any timeline analysis to determine if there's a specific point(s) within our basic test where we start to see packet loss or whether the loss was uniform.

I do recall reading a vendor produced whitepaper, where circa 900 mesh devices were deployed over a 2,000 sq m. area with a target response time/latency of 300 msec being set for message delivery in message traffic configs of (low - ~150bps) - (medium - ~1kbps) - (high - ~3kbps). I grabbed a screenshot of the results at the time (see below), which highlighted that at least 99.1% of messages were successfully delivered with 300 msec latency across a high traffic (enhanced configuration) in both sparse and dense deployments. I recall that enhanced meant that source message re-transmissions were performed as well as advertising randomization whereas Baseline was a standard mesh implementation. Also note that this wasn't Nordics's mesh stack!! (If you are interested, let me know and I'll see if I can dig out the link to the whitepaper.)

I guess the general takeaway(s) from the above would be:

that we shouldn't base any firm design decision(s) on the packet loss figures that I provided (since the config is/was not optimized or validated)
the High traffic w/Dense deployment Baseline stat (69.2% - from the table), does reflect a similar level of degradation to what we saw in our basic test
there are Enhanced optimizations that can be implemented to increase mesh reliability and throughput
the results reflect fairly well on Bluetooth mesh when compared to results i've seen across Zigbee, Z-wave etc. based on lower device numbers (the caveat being that this is only one specific configuration under test)

I hope the above helps.

Regards,

RE: Need help to improve Bluetooth Mesh throughput

aleccontrol — Fri, 16 Mar 2018 05:06:41 GMT

Hi Leonwj,

You are right that I was looking at the wrong "case" statement so yes, the following statement was indeed the default for "case :PROV_STATE_CONFIG_PUBLICATION_HEALTH"

pubstate.publish_period.step_res = ACCESS_PUBLISH_RESOLUTION_10s

Thank you very much for testing out the command/response program. I think the large percentage of packet losses at the server is probably the reason why we feel that the communication appears to choke when attempt to send out faster than 2 packets a second.

In an M2M communication typically when the client sends out a command packet, it will wait for a response packet to be received from the server before it proceed to send the next command. If the server fails send a response, then the client will time out after some time (in our case we set the time out to 1 second). The client will only send the next command packet after the time out.

If 25% of the consecutive commands is lost as per your test, that means that the client will fail to receive 1 in 4 responses. The client is thus unable to sustain a smooth command/response communication with the server every 100ms as it constantly have to wait for the time out before it can send the next packet. So the communication appears to stop frequently due to the time out.

What is the reason for such high packet loss and is there any way to minimize the packet loss? The client and the server are actually sitting at less than 1 meter apart and I only have 1 client and 1 server communicating. There wasn’t any other Bluetooth devices communicating nearby. What would be the maximum command/response exchange rate in order to maintain <= 1% packet loss?

Thanks again for your expert advise!

RE: Need help to improve Bluetooth Mesh throughput

leonwj — Thu, 15 Mar 2018 08:18:57 GMT

hello aleccontrol,

I just re-downloaded both the v1.0 and v1.01 Mesh SDKs from the Nordic website and they both have the case PROV_STATE_CONFIG_PUBLICATION_HEALTH: set to ACCESS_PUBLISH_RESOLUTION_10S. (Possibly you are confusing this with the case PROV_STATE_CONFIG_PUBLICATION_ONOFF: case which is directly below in the code). So i'm not sure if that setting pertains to your issue.

In any case, I went ahead and implemented the simple_on_off model that you attached within our environment. As you suggested, we commented out the UART comm1_strout functionality and simply echoed back out the received command string along with a counter value to iterate through the responses from each server node. We set the send_command_client(&m_clients[GROUP_CLIENT_INDEX], command, 4); function to trigger every time we received a heartbeat message from any of the 3 provisioned nodes.

So in the above scenario, the 3 server nodes send out a heartbeat message (at the specified interval, we tested 1 sec and 0.1 sec) which then triggers the client node to send the "Test" command to the 3 servers which each then send back an echoed response.

Needless to say, we are not seeing the same throughput issue that you have reported. I have attached the RTT log file from the 100MS (0.1 sec) test run that we analyzed over a 5 minute (300 sec) period. Within that log we see significant mesh network throughput (albeit a simple echo of the "Test" string with a counter value returned). We do however see approximately 25% packet loss at that message frequency.

Regards,

devzone.nordicsemi.com/.../rtt_2D00_log_2D00_100ms_2D00_cmd_2D00_resp_5B00_3_2D00_svrs_5D00_.zip

RE: Need help to improve Bluetooth Mesh throughput

aleccontrol — Thu, 15 Mar 2018 07:06:09 GMT

Hello Leonwj,

Thank you very much for the suggestion.

The BLE Mesh version 1.0.0 that I have downloaded already contain this line in the

case PROV_STATE_CONFIG_PUBLICATION_HEALTH:

pubstate.publish_period.step_res = ACCESS_PUBLISH_RESOLUTION_100MS;

Does that mean the stack I have been working on is already flooding the comm channel with heartbeat check every 0.1s? Could that be the reason why the intended command/response packet is being affected by the rapid heart beat check?

I tried changing it back to 10s (I assume that is what it should have been) but I am still experiencing the same latency issue. Do I need to re-provision the client and the server for the heart beat period to change?

By the way is there a quick way force the client to re-provision the server? So far I have used "Erase All Memory" option to clear the provisioning data on both clients and servers so that a new provisioning can take place.

RE: Need help to improve Bluetooth Mesh throughput

leonwj — Wed, 14 Mar 2018 20:51:37 GMT

hello aleccontrol,

We have also setup a command/control model to test Bluetooth mesh throughput/reliability, however we have not experienced the issue(s) that you mentioned. I'm not clear on the use case that you are trying to implement which requires the mesh client model to send messages with that level of frequency but would suggest that in order to test throughput across your mesh environment, do the following...

Bluetooth Mesh implements a Health server as a root model so by utilizing that you can quickly monitor message delivery across the mesh network. Using the default light-switch example (with 1 x client and 3 x servers) amend provisioner.c in the client to request heartbeat messages every 100 milliseconds as follows (note: the default 10s heartbeat has been commented out):

        /* Configure the publication parameters for the On/Off server: */
        case PROV_STATE_CONFIG_PUBLICATION_HEALTH:
        {
            config_publication_state_t pubstate = {0};
            pubstate.element_address = m_target_address;
            pubstate.publish_address.type = NRF_MESH_ADDRESS_TYPE_UNICAST;
            pubstate.publish_address.value = PROVISIONER_ADDRESS;
            pubstate.appkey_index = 0;
            pubstate.frendship_credential_flag = false;
            pubstate.publish_ttl = SERVER_COUNT;
            pubstate.publish_period.step_num = 1;
            //pubstate.publish_period.step_res = ACCESS_PUBLISH_RESOLUTION_10S;       // WJL
            pubstate.publish_period.step_res = ACCESS_PUBLISH_RESOLUTION_100MS;
            pubstate.retransmit_count = 1;
            pubstate.retransmit_interval = 0;
            pubstate.model_id.company_id = ACCESS_COMPANY_ID_NONE;
            pubstate.model_id.model_id = HEALTH_SERVER_MODEL_ID;
            __LOG(LOG_SRC_APP, LOG_LEVEL_INFO, "Setting publication address for the health server to 0x%04x\n", pubstate.publish_address.value);

            ERROR_CHECK(config_client_model_publication_set(&pubstate));
            break;
        }

This will obviously request each provisioned server to send 10 messages a second across the mesh, with the expectation that the 3 servers would push 30 such heartbeats each second. Based upon the results, both reliability and throughput can be measured. The results we see in our environment are as follows:

So across a 4 minute and 2 second time-span (242 seconds), we're seeing 7,200 messages across the mesh which equates to approx. 30 per sec. which in turn is reflective of the 30 heartbeats that we expected (The mesh spec limits each node to 24 Hz so this is within spec)

You could additionally amend your environment by increasing the distance between the server nodes and the base client (and adding simple relay nodes where required). Again, the above doesn't mimic your command/response model but it does allow you to at least verify mesh throughput with a single line change to the basic example code.

I'll try to implement your model in our environment and post back if I see any extraneous results.

Regards,

RE: Need help to improve Bluetooth Mesh throughput

David Edwin — Wed, 14 Mar 2018 13:59:32 GMT

statistical information is available through scanner.h -> scanner_stats_get() , this should get you the statistics for the normal(excludes instaburst) mesh channel operations.

Instaburst statistics are available through instaburst_rx.h -> instaburst_rx_stats_get(channel) , you will need to iterate through all the channels you are using.

RE: Need help to improve Bluetooth Mesh throughput

vishal — Wed, 14 Mar 2018 10:46:02 GMT

Hi,

I am also trying to send text message over ble mesh but until not find solution.

Can you provide me your light switch main.c file. I have download the above .zip file but try to send simple test message but this also not send. Please help me for send text message client to server.

Thanks..

RE: Need help to improve Bluetooth Mesh throughput

David Edwin — Wed, 14 Mar 2018 10:38:27 GMT

Segmentation And Reassembly

RE: Need help to improve Bluetooth Mesh throughput

au — Wed, 14 Mar 2018 09:03:12 GMT

What is SAR here?

RE: Need help to improve Bluetooth Mesh throughput

David Edwin — Wed, 14 Mar 2018 07:53:15 GMT

The fastest transfer methods are unreliable and where the application manages the responses and not the lower stack. It appears you are doing this. However verify that you are not using a reliable message. (I have not checked the zip file as I noticed it quite late)

Edited : 4 packets a second is within the capability of the Mesh. 24-20 packets per seconds is possible to be transmitted but there will be a portion of packets lost based on topology. You many need to consider the Nordic Instaburst as well if you want larger bandwidth

Get the statistical data from the stack so you can check if your configuration of queue sizes are ok. i.e. no overflows etc.

You can also check the return codes of the functions where you are sending data to the stack to see if did manage to get to the incoming queues.

RE: Need help to improve Bluetooth Mesh throughput

aleccontrol — Wed, 14 Mar 2018 06:39:57 GMT

Thanks I did try with sending command message that is only 4 bytes long and got response that is also 4 bytes long. It has improved a bit when tried to send two commands every seconds but the communication still chokes when tried to increase the frequency to sending about 4 commands a second. When the communication chokes you can see that the server simply stop responding for a few seconds and then handle the last command it receives.

We are hoping to be able to exchange text messages of possibly between 10 to 100 bytes via BLE Mesh, just like how we normally do via UART. Many control applications make use of such command/response messages to exchange data between devices (e.g. Modbus ASCII or RTU). BLE Mesh is supposed to be able to handle data payload as large as 300+ bytes, but even when we tested with small packets such as 10 to 15 bytes the delay we experienced frankly is unacceptably high to be useable for command/response message exchange. I hope what I encountered is due to my configuration and not something inherent to BLE mesh.

RE: Need help to improve Bluetooth Mesh throughput

aleccontrol — Wed, 14 Mar 2018 06:20:29 GMT

devzone.nordicsemi.com/.../simple_5F00_on_5F00_off.zip

Thanks for your reply.

I have attached the modified files for the "simple_on_off_client", "simple_on_off_server" used for testing.

Client

To test the command/response, add the following code fragment to the main.c in the simple_on_off_client project:

   extern uint32_t send_command_client(simple_on_off_client_t * p_client, uint8_t *command, uint16_t length);

   uint8_t command[] = "Test";
        status = send_command_client(&m_clients[GROUP_CLIENT_INDEX], command, 4);

Note: I have edited the length to 4 (was 9 when first posted) since the command string now only contains 4 bytes)

Server

The simple_on_off_server has been modified so that it will simply copy the message it received and echo back to the client as a response package.

The client will run the function "handle_response_cb() when it receives the response from the server. You can log the response it received from the server. In our program we actually send the message out of the UART that requires to link with the app_uart files so you can simply comment out the call to "com1_strout() in our program".

Thank you if you can share with us your finding and suggested solutions.