nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Hung Bui — Fri, 06 May 2022 10:53:06 GMT

Hi Stanley,

As far as I can see increasing CORE_TX_QUEUE_BUFFER_SIZE_RELAY wouldn't cause any side effect except for that it would require more RAM and maybe making the latency for sending the originator message longer (just my guess). It's a fixed buffer so this amount of RAM is dedicated for CORE_TX_QUEUE_BUFFER_SIZE_RELAY.

My suggestion is to try CORE_TX_QUEUE_BUFFER_SIZE_RELAY at 512 but with only one sendmessage() call at a time. Then you can run a test and check the success rate to decide how many times should you resend.

2. I don't think CORE_TX_QUEUE_BUFFER_SIZE_ORIGINATOR would help. It's for the message that's originated from the proxy node.

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Stanley Hung — Fri, 06 May 2022 09:29:27 GMT

Thank you for your information, and I increased
- CORE_TX_QUEUE_BUFFER_SIZE_RELAY to 512 Bytes, and
- sent 100 bytes (resend 4 times) to group address,
The other nodes receive packet successfully sometimes, and sometimes fails. In this condition, increasing connection interval is not improved obviously.

1) In general, if increasing the RAM buffer in CORE_TX_QUEUE_BUFFER_SIZE_RELAY, to the upmost of a processor, what will be the disadvantage? Or what is the best percentage of RAM size to be allocated for buffer (I assume after proxy command, the RAM will be released, is it?)

2) Will it help by increasing CORE_TX_QUEUE_BUFFER_SIZE_ORIGINATOR in core_tx_adv.c?

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Hung Bui — Thu, 05 May 2022 13:10:48 GMT

Hi Stanley,

I believe the buffer that you may want to try to increase is CORE_TX_QUEUE_BUFFER_SIZE_RELAY (128) in core_tx_adv.c

You can find how it's called by looking at rx_handle() in proxy.c where the MESH_GATT_PDU_TYPE_NETWORK_PDU is feed into network_packet_in().

Inside network_packet_in() packet_relay() will be called if it's the packet to be relayed. And inside that you can find that if the buffer is full you will simply get a Warning level log of "Unable to allocate memory for relay packet" and the packet will be discarded.

So I believe that if the data to be relayed are queued larger than CORE_TX_QUEUE_BUFFER_SIZE_RELAY they will be just discarded. This is fine if it's normal relay packet as the packet may get relayed by other node. But if it's the proxy node is the only node that has the packet to be relayed it will be discarded.

This is different from when the proxy node itself want to send a large SAR message. The segmented message will be sent one by one from the source and the relay buffer is not used (because it's the source, not the relay).

So what you can do is to try increasing CORE_TX_QUEUE_BUFFER_SIZE_RELAY, and also try to increase the connection interval between the phone and the proxy node, so that the data throughput is reduced giving more time for the proxy node to relay the messages.

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Stanley Hung — Thu, 05 May 2022 08:49:52 GMT

In our application, the larger number of packets, the better application performance (in terms of user experience). Before this issue is fixed, we will try to find an optimum condition to make the design works. It would be very much appreciated if you could keep me informed once a new solution is available.

Could you share me more details about “the buffer for all segments” in the communication between phone and proxy node? Perhaps, I can do some test accordingly.

By the way, I also summarize what I found these days for your reference:

It requires 4 times of retry number (TRANSPORT_SAR_TX_RETRIES_DEFAULT (4)) on phone to send a group message via a proxy node. And the message size is limited to 11 Bytes with successful sending.

In above condition, when the packet size is over 11 Bytes for group address, it always fails.

In case sending a group message from a nRF52840DK (generic onoff client), the result is successful even the packet size is up to 100 Bytes. Why not from a phone ?

With unicast address from a phone to a node via a proxy node, and the retry number is set to 4 times, the packet size can be up to 100 Bytes. However, the performance is poor in sending time.

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Hung Bui — Wed, 04 May 2022 14:20:13 GMT

Hi Stanley,

We are suspecting that the issue may be due to the communication between the phone and the proxy node doesn't have enough buffer for all segments. We are looking into it.

If you are not planning to send the data to large number of node I think sending it to unicast address would be a better solution. SAR message when sending to unicast address will have acknowledgment on transport layer so you can be sure the data has arrived to the end node properly.

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Stanley Hung — Wed, 04 May 2022 09:57:58 GMT

For your information, I tried sending a unicast address with 100 Bytes, it works, The only thing I will beef about it takes a longer time to complete it.

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Hung Bui — Tue, 03 May 2022 11:12:31 GMT

Hi Stanley,

1. Could you please elaborate what you meant by "to look up the “time slot” on the proxy node" ?

2. I'm also not fully understand what you meant by limiting the segment. Please clarify.

3. As far as I know they are not related. A proxy node use GATT bearer and relay the message between the phone and the mesh network. It only relays when the message is targeting or from the phone. When the relay node (or relay role) is to relay the any mesh message, from ATT bearer to ATT bearer.

The difference between sending 100 bytes to one node compare to sending 100 bytes to mutliple node (group address) is that there is no acknowledgement when you send the segmented message to a group address. Here the quote from the Mesh spec:

This explain why sending the same data to a single node result successful when sending the same message to a group message failed.

However I haven't been able to explain why the performance is lower when sending (with multiple retries) from the phone compare to sending directly from a node in the network.

If we couldn't find a root cause for this you may have to consider offload the task of sending the large chunk of data to the proxy node. This can be done by implementing proprietary code on the proxy node.

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Stanley Hung — Tue, 03 May 2022 06:31:25 GMT

Yes, By call “sendmessage()” 4 times with 12 Bytes payload size, the message is successfully arrived at other nodes in the mesh network. However, it fails if the payload is increased to 100 Bytes.

Please be reminded that
- In case sending 100 Bytes to ONE node from smartphone, the result is successful.
- Sending 100 Bytes from nRF52840DK to another 2 mesh nodes is successful.

Please comment:
1. We might still need to look up the “time slot” on the proxy node if the payload length is over some certain bytes?

2. How if we improve this issue by limit the “segment” on the proxy node?

3.While relaying a message from one node to another, is there any difference between a proxy node and relay node in this application?

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Hung Bui — Mon, 02 May 2022 10:26:33 GMT

Hi Stanley,

I had a discussion with the team here about the issue. They pointed out that on normal node (not the phone) the segmented multicast message has a retry number TRANSPORT_SAR_TX_RETRIES_DEFAULT (4) to resend the segmented messages and make sure they would arrive to the destinations.

It's not the case on the phone as sendmessage is called only one.

Could you try to do a test to send a message of , for example 12 bytes, and try to resend it (call sendmessage()) 4 times and check if it's arrived on the other nodes in the mesh network ?

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Stanley Hung — Fri, 29 Apr 2022 08:45:17 GMT

Thank you for your reply.

In this model, I use sendVendorModelMessage to send my packet to specified group address and store the meshmessage as a VendorModelMessageAckedState.

I have tried 30 bytes , but the other nodes could not received.

It will occur as long as the payload exceeds 8 bytes.

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Hung Bui — Fri, 29 Apr 2022 07:59:51 GMT

Hi Stanley,

As far as I know the Proxy node should be able to do segmented relaying.

Could you show exactly how you send the message from the code in your app ?

Have you tried to send something smaller, for example 30 bytes ? To see if it's received by all nodes. We just need to know if it was the issue with larger message size or it's the issue with any segmented message.

I think there could be an issue when the source device need to resend a segmented message but it's connected via Proxy so there is a chance that the proxy couldn't relay the ACK and the re-transmission package.

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Stanley Hung — Fri, 29 Apr 2022 07:19:57 GMT

I believe I am using “group” as you mentioned instead of “broadcast”, since I use a comment (in Android-nRF-Mesh-Library) to send my packet to specified group address.

Furthermore, a similar test is conducted by 3 of nRF52840DK, where

- No.1 works as a generic onoff client;

- No.2 works as a server, and is placed in the wireless working rang with No.1; and

- No.3 works as a server, and placed far away from No1, but in the working range of No.2.

The result is No.1 is able to publish 300 bytes to both No2 and No3 successfully by nrf mesh publish and subscribe addresses.

My question is:

Does it indicate that the proxy node has any relay restrictions?

RE: nRF Mesh , sending hundreds of bytes packets to the mesh nodes causes NRF_MESH_EVT_SAR_FAILED

Hung Bui — Thu, 28 Apr 2022 10:36:04 GMT

Hi Stanley,

Could you please describe what exactly you did by doing "broadcasting" from the phone?

You are correct that the phone only connect to a proxy node to talk to the mesh network. Usually it's the last provisioned node. You can disconnect a proxy node and connect to other proxy node of your choice.

Please be aware that in most case a phone doesn't do mesh directly. It requires a BLE connection to a proxy node to be able to talk to other nodes.

There isn't a "broadcast" packet in mesh, you can send a packet to all nodes (address 0xFFFF) but it's not recommended. It's preferred if you setup a Group address and then subscribe the model on the nodes to this address.

Also, Bluetooth mesh is not designed for high throughput communication. So sending a large amount of bytes to all the nodes in the network expose the risk of network congestion as packet about 11bytes will be segmented. For example to transmit 100 bytes, there would be 12 segmented packets, this also require ACK packets for each of them.