Odd/Unexpected Behavior from NRFMesh

[NickN]

Greetings.

We've been developing a product that uses BLE Mesh. We are currently using the Waveshare nRF52840 Bluetooth 5.0 Evaluation Kit for prototyping purposes. As far as I know, we are using the latest SDK and our firmware is based off the "sensor" project example.

I am seeing two unexpected behaviors with nRFMesh. I would welcome any feedback you all might have.

Issue #1

I currently have seven nodes provisioned and reporting to a gateway. Five are definitely within range of the gateway. One is definitely out of range. One is right on the border (sometimes it seems to report directly and other times the message seems to get relayed).

I am provisioning the mesh using an Android phone and nRFMesh. We have two custom element models defined in the node server profile (a server model and a setup model). The server model is set to publish to a group address every 10s (the publish period is set to 10s). Within "Publication Settings" for the element model that is publishing messages, I set the "Publish TTL" value manually to 15.

Here's what's odd: when I come back to review the mesh settings at a later time, I often find that TTL has reset to a value of 1. Please note that I am connected to the mesh within nRFMesh when I check this value. However, I am still receiving messages at the gateway, so the data must be being relayed. i.e. the TTL cannot be 1.

Equally confusing, the default TTL on the node configuration page of the app still shows a value of 15.  

Is this normal behavior? It seems odd that the value is reset to 1 when it clearly cannot be 1.

Issue #2

Let's say I want to update the Publication Settings for the nodes in my mesh. I connect to the mesh within the nRFMesh app and then change the parameters for each provisioned node.

This works fine for nodes that are definitely within range of wherever I am at the time. I can connect to any of the five nodes in range and make changes to any of those five without reconnecting/changing which node I connect to within nRFMesh. 

Here's what's odd: For the two nodes that are out of range, I have to move to be within their range and connect to the node I want to change within nRFMesh. 

Since this is a mesh and data is being relayed, I would expect to be able to change the settings of any provisioned node in the mesh, regardless of its location (as long as it is connected to another node).

 

My concern with both issues is that we may have something set incorrectly, or some underlying bug we need to squash. All suggestions/comments are most welcome.

 

[daviddedwin]

#1 For the TTL issue, use the nRF Sniffer and take a look at the messages that are actually on the network. It maybe an Android app issue.
However posting the nRFSniffer wireshark capture file will be useful.
Additionally post the Android app version and the Android OS version that you are using.

#2 Take a look at the relay settings for the nodes, ideally all nodes should not be relaying as that will choke the network however you should look at the nRFMesh setting for Relay and also in the code as the Relay capability can be set both ways.

Actions requested:
- nRFSniffer Wireshark trace
- Android app version
- Android OS version
- Post a map of the network 
- Post the JSON file that you have exported from the nRFMesh

[NickN]

Thanks for the quick reply.

- nRFSniffer Wireshark trace

Work in progress. I've set up nRFSniffer with an nRF Dongle, but I'm new to wireshark so it will take me a short while to figure out how to get useful data out of it. Working with my developer to figure this out.

- Android app version

3.1.5

- Android OS version

11, running on a Samsung S21 Ultra

- Post a map of the network 

- Post the JSON file that you have exported from the nRFMesh

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://www.bluetooth.com/specifications/assigned-numbers/mesh-profile/cdb-schema.json#",
  "version": "1.0.0",
  "meshUUID": "4076B36A-9332-495F-8AA8-A270EEEA5EB9",
  "meshName": "nRF Mesh Network",
  "timestamp": "2021-05-02T15:25:54-04",
  "partial": false,
  "netKeys": [
    {
      "name": "Network Key 1",
      "index": 0,
      "key": "759738F46882F31FEC9887E4A23BDE99",
      "phase": 0,
      "minSecurity": "insecure",
      "timestamp": "2021-05-02T14:30:21-04"
    }
  ],
  "appKeys": [
    {
      "name": "Application Key 1",
      "index": 0,
      "boundNetKey": 0,
      "key": "DA7979E5B38D4525F9B4FD1AFF4A5294"
    },
    {
      "name": "Application Key 2",
      "index": 1,
      "boundNetKey": 0,
      "key": "381ED8CC5E001DCE617E3615A584739B"
    },
    {
      "name": "Application Key 3",
      "index": 2,
      "boundNetKey": 0,
      "key": "E61A3AE2D95C1E7FE5AB9CEFE3B5C95B"
    }
  ],
  "provisioners": [
    {
      "provisionerName": "nRF Mesh Provisioner",
      "UUID": "B48C3296-6F72-405A-AB48-B1A3F0C7CAAC",
      "allocatedUnicastRange": [
        {
          "lowAddress": "0001",
          "highAddress": "199A"
        }
      ],
      "allocatedGroupRange": [
        {
          "lowAddress": "C000",
          "highAddress": "CC9A"
        }
      ],
      "allocatedSceneRange": [
        {
          "firstScene": "0001",
          "lastScene": "3333"
        }
      ]
    }
  ],
  "nodes": [
    {
      "UUID": "B48C3296-6F72-405A-AB48-B1A3F0C7CAAC",
      "name": "nRF Mesh Provisioner",
      "deviceKey": "B2A5F637F59705B97762FE493C46A7FB",
      "unicastAddress": "0001",
      "security": "insecure",
      "configComplete": true,
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 2,
        "relay": 2
      },
      "defaultTTL": 5,
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        },
        {
          "index": 1,
          "updated": false
        },
        {
          "index": 2,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0001",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0001",
              "bind": [],
              "subscribe": []
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "3A7AD8F9-4C5F-45E0-98AC-8580EBF9835C",
      "name": "Gateway 002 AC:D8",
      "deviceKey": "9CDB9E1D4405621574830EF009537448",
      "unicastAddress": "0002",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 40,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0002",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            }
          ]
        },
        {
          "name": "Element: 0x0003",
          "index": 1,
          "location": "0000",
          "models": [
            {
              "modelId": "1102",
              "bind": [
                0
              ],
              "subscribe": [
                "C000"
              ],
              "publish": {
                "address": "C010",
                "index": 0,
                "ttl": 1,
                "period": {
                  "numberOfSteps": 0,
                  "resolution": 100
                },
                "retransmit": {
                  "count": 3,
                  "interval": 100
                },
                "credentials": 0
              }
            }
          ]
        },
        {
          "name": "Element: 0x0004",
          "index": 2,
          "location": "0000",
          "models": [
            {
              "modelId": "1102",
              "bind": [],
              "subscribe": []
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "962CDE69-8BC4-413F-ACDE-86BB662C7975",
      "name": "0005 DE:EC",
      "deviceKey": "3D6B0477DA08D1F1745D06256D428154",
      "unicastAddress": "0005",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 15,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0005",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 1,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 1,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "F737B90F-FED8-42AA-87E9-A0CD3AA7E3B1",
      "name": "0006 E9:07",
      "deviceKey": "2327B46BD9E6BC62FFD6186FAC1B5CDA",
      "unicastAddress": "0006",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 15,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0006",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 1,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 1,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "38BC8408-FD73-43B4-9877-EFD9E04035EF",
      "name": "0007 77:18",
      "deviceKey": "B82AC438F5FCA510CB45C86A08F869BA",
      "unicastAddress": "0007",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 15,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0007",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 1,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 1,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "B7774E19-91DA-4B32-8340-59D78553445D",
      "name": "0008 40:03",
      "deviceKey": "434F0060A3D1399D4B93E77558E5962D",
      "unicastAddress": "0008",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 15,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0008",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 1,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 1,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "70B2CBD9-C85F-49B1-8F1B-C7F38859FC68",
      "name": "0009 1B:CF",
      "deviceKey": "A98420C24D35056627F15CE54130276C",
      "unicastAddress": "0009",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 15,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0009",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 1,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 1,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "7398B249-7CE5-4BA3-BCD5-E242882A1C11",
      "name": "Dead",
      "deviceKey": "E8D823F3C8C0411AAA614309D4F6CD0A",
      "unicastAddress": "000A",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 15,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x000A",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 1,
                "period": {
                  "numberOfSteps": 50,
                  "resolution": 100
                },
                "retransmit": {
                  "count": 1,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "03B4796F-5D05-4357-8249-3E3266F0BE0B",
      "name": "000C 49:42",
      "deviceKey": "AC4E9ACA299C91DB37DDDC578D5FCF2F",
      "unicastAddress": "000C",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 15,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x000C",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 1,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 1,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "AB89D50B-7FDE-4E41-84A7-75BAE822FA1A",
      "name": "000D A7:44 Good",
      "deviceKey": "FAAAF943E35E15541453E2CD304D537F",
      "unicastAddress": "000D",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 15,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x000D",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 1,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 1,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    }
  ],
  "groups": [
    {
      "name": "SensorData",
      "address": "C000",
      "parentAddress": "0000"
    },
    {
      "name": "TimeSync",
      "address": "C010",
      "parentAddress": "0000"
    }
  ],
  "scenes": [],
  "networkExclusions": []
}

[daviddedwin]

Thank you, this starts to show some things.

1. The yellow node and and the red node where the connectivity is poor will need to be sniffed to see if they are actually getting relayed, they are in the edge of the range.
 - Followup questions and actions
   What is your transmit power, set this to the maximum that it possible.
   What is your antenna type and external antenna is preferred here.

2. The TTL numbers for "defaultTTL" are all over the place, for such a small network try setting it to TTL=5 but set the retransmit to at least 3 for all node transmissions. A re-transmit setting of 1 will not work and will cause packet losses.

3. What is the bandwidth on this network, can you comment on the message rates between nodes and gateways.

My first hypothesis is that the relaying is not working well due to the low retransmit settings so that will be the main thing to check.

My second hypothesis is that there are proxies running on all the nodes, please provide the advertising interval and the connection interval being used on the proxy, this can also choke the relaying capacity and show symptoms like the ones you are seeing.

 

[NickN]

Thanks again for the help. Some updates.

1. The yellow node and and the red node where the connectivity is poor will need to be sniffed to see if they are actually getting relayed, they are in the edge of the range.
 - Followup questions and actions

   What is your transmit power, set this to the maximum that it possible.

It was +2dB, but we've now set it to +8dB.

   What is your antenna type and external antenna is preferred here.

It's an internal antenna. We're using an off the shelf module from EByte (this one: https://lcsc.com/product-detail/Bluetooth-Modules_Chengdu-Ebyte-Elec-Tech-E73-2G4M08S1C_C356849.html)

2. The TTL numbers for "defaultTTL" are all over the place, for such a small network try setting it to TTL=5 but set the retransmit to at least 3 for all node transmissions. A re-transmit setting of 1 will not work and will cause packet losses.

Using nRF mesh, I created a new mesh and used retransmit of 3, default TTL of 5 and element model TTL of 5. Here's an updated mesh file -- let me know if it looks better in terms of TTL settings etc.

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://www.bluetooth.com/specifications/assigned-numbers/mesh-profile/cdb-schema.json#",
  "version": "1.0.0",
  "meshUUID": "363C475A-5A23-4116-A86C-95BB6921052F",
  "meshName": "nRF Mesh Network",
  "timestamp": "2021-05-09T17:23:47-04",
  "partial": false,
  "netKeys": [
    {
      "name": "Network Key 1",
      "index": 0,
      "key": "E4497AADB85091B2150132FCCC32BDAD",
      "phase": 0,
      "minSecurity": "insecure",
      "timestamp": "2021-05-09T15:26:37-04"
    }
  ],
  "appKeys": [
    {
      "name": "Application Key 1",
      "index": 0,
      "boundNetKey": 0,
      "key": "A46F6B8E48F33476C019F719FAEC0C7E"
    },
    {
      "name": "Application Key 2",
      "index": 1,
      "boundNetKey": 0,
      "key": "6D78DEBA6F28202D78FB234844305E55"
    },
    {
      "name": "Application Key 3",
      "index": 2,
      "boundNetKey": 0,
      "key": "8E3BE85A2D2FAF6647AE599280F3B800"
    }
  ],
  "provisioners": [
    {
      "provisionerName": "nRF Mesh Provisioner",
      "UUID": "43AF85C4-9697-4D1C-B383-72885F09DD8C",
      "allocatedUnicastRange": [
        {
          "lowAddress": "0001",
          "highAddress": "199A"
        }
      ],
      "allocatedGroupRange": [
        {
          "lowAddress": "C000",
          "highAddress": "CC9A"
        }
      ],
      "allocatedSceneRange": [
        {
          "firstScene": "0001",
          "lastScene": "3333"
        }
      ]
    }
  ],
  "nodes": [
    {
      "UUID": "43AF85C4-9697-4D1C-B383-72885F09DD8C",
      "name": "nRF Mesh Provisioner",
      "deviceKey": "853E2B4D9CFEF32E28E176DFA68A5027",
      "unicastAddress": "0001",
      "security": "insecure",
      "configComplete": true,
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 2,
        "relay": 2
      },
      "defaultTTL": 5,
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        },
        {
          "index": 1,
          "updated": false
        },
        {
          "index": 2,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0001",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0001",
              "bind": [],
              "subscribe": []
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "3A7AD8F9-4C5F-45E0-98AC-8580EBF9835C",
      "name": "AC:D8 Gateway 002",
      "deviceKey": "13AD4EA1BD11B9822BB4A83D9D79137F",
      "unicastAddress": "0002",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 10,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0002",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            }
          ]
        },
        {
          "name": "Element: 0x0003",
          "index": 1,
          "location": "0000",
          "models": [
            {
              "modelId": "1102",
              "bind": [
                0
              ],
              "subscribe": [
                "C000"
              ],
              "publish": {
                "address": "C010",
                "index": 0,
                "ttl": 5,
                "period": {
                  "numberOfSteps": 0,
                  "resolution": 100
                },
                "retransmit": {
                  "count": 3,
                  "interval": 100
                },
                "credentials": 0
              }
            }
          ]
        },
        {
          "name": "Element: 0x0004",
          "index": 2,
          "location": "0000",
          "models": [
            {
              "modelId": "1102",
              "bind": [],
              "subscribe": []
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "962CDE69-8BC4-413F-ACDE-86BB662C7975",
      "name": "DE:EC Sensor 005",
      "deviceKey": "F90D89B971C351790198934F4355CF28",
      "unicastAddress": "0005",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 5,
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0005",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 5,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 3,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "F737B90F-FED8-42AA-87E9-A0CD3AA7E3B1",
      "name": "E9:07 Sensor 006",
      "deviceKey": "86D52ED707529B20702E744E99E27004",
      "unicastAddress": "0006",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 5,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0006",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 5,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 3,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "38BC8408-FD73-43B4-9877-EFD9E04035EF",
      "name": "77:18 Sensor 007",
      "deviceKey": "1846BD69D4377D9FE1EB0D37215DE933",
      "unicastAddress": "0007",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 5,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0007",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 5,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 3,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "B7774E19-91DA-4B32-8340-59D78553445D",
      "name": "40:03 Sensor 008",
      "deviceKey": "5F6DDB88D8B77DF58365575F61556BB6",
      "unicastAddress": "0008",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 5,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0008",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 5,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 3,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "70B2CBD9-C85F-49B1-8F1B-C7F38859FC68",
      "name": "1B:CF Sensor 009",
      "deviceKey": "186558F1BFA4C46BE49A4B676F2C402E",
      "unicastAddress": "0009",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 5,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0009",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 5,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 3,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "7398B249-7CE5-4BA3-BCD5-E242882A1C11",
      "name": "D5:3C Sensor 00A",
      "deviceKey": "C6F460E3CEACD1AB7254A2F4282BD131",
      "unicastAddress": "000A",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 5,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x000A",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 5,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 3,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "AB89D50B-7FDE-4E41-84A7-75BAE822FA1A",
      "name": "A7:44 Sensor 00B",
      "deviceKey": "9F1EB4DD5B08E035DEEA6D39F7520E67",
      "unicastAddress": "000B",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 5,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x000B",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 5,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 3,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "03B4796F-5D05-4357-8249-3E3266F0BE0B",
      "name": "49:42 Sensor 00C",
      "deviceKey": "115F8EA86957E71658D2A335951AE13A",
      "unicastAddress": "000C",
      "security": "insecure",
      "configComplete": false,
      "cid": "0059",
      "pid": "0000",
      "vid": "0000",
      "crpl": "0028",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 5,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x000C",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "0002",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1100",
              "bind": [
                0
              ],
              "subscribe": [],
              "publish": {
                "address": "C000",
                "index": 0,
                "ttl": 5,
                "period": {
                  "numberOfSteps": 10,
                  "resolution": 1000
                },
                "retransmit": {
                  "count": 3,
                  "interval": 100
                },
                "credentials": 0
              }
            },
            {
              "modelId": "1101",
              "bind": [
                0
              ],
              "subscribe": [
                "C010"
              ]
            }
          ]
        }
      ],
      "excluded": false
    }
  ],
  "groups": [
    {
      "name": "SensorData",
      "address": "C000",
      "parentAddress": "0000"
    },
    {
      "name": "TimeSync",
      "address": "C010",
      "parentAddress": "0000"
    }
  ],
  "scenes": [],
  "networkExclusions": []
}

3. What is the bandwidth on this network, can you comment on the message rates between nodes and gateways.

Messages are being sent every 10 seconds and right now there are 8 nodes plus a gateway. The messages are 23 bytes each, plus the headers, CRC etc that the mesh stack attaches.

In the code, we are using the 1 Mb data rate (RADIO_MODE_BLE_1MBIT).

My first hypothesis is that the relaying is not working well due to the low retransmit settings so that will be the main thing to check.

My second hypothesis is that there are proxies running on all the nodes, please provide the advertising interval and the connection interval being used on the proxy, this can also choke the relaying capacity and show symptoms like the ones you are seeing.

Here's what our lead engineer sent regarding advertising interval etc. I gather they are defined in mesh\core\api\nrf_mesh_config_core.h

/**
 * Advertisement interval for Mesh GATT Proxy Network ID advertisements.
 */
#ifndef MESH_GATT_PROXY_NETWORK_ID_ADV_INT_MS
#define MESH_GATT_PROXY_NETWORK_ID_ADV_INT_MS 2000
#endif

So 2 second interval for advertising the network ID.

and

/**
 * Advertisement interval for Mesh GATT Proxy Node Identity advertisements.
 *
 * The Node Identity beacon is used by the Provisioner to identify a specific node with Proxy
 * support. The beacon is automatically advertised after the node has been provisioned,
 * if the node supports the Proxy feature.
 */
#ifndef MESH_GATT_PROXY_NODE_IDENTITY_ADV_INT_MS
#define MESH_GATT_PROXY_NODE_IDENTITY_ADV_INT_MS 200
#endif

i.e. a 2 second interval for proxy ID advertisements.

Proxy Settings in NRF Mesh 

I'm finding the nRF Mesh app a bit confusing here. If I click "read state" I see red text that says "OFF". This implies to me that the proxy state is off. However, the text above it says "Turning off the proxy feature will set the node to a non-connectable state and disable the proxy server.". This rather implies that tapping the red text will turn the proxy off i.e. it is currently on.

If I look at the node details page in the app, Relay Feature and Proxy Feature both show as enabled.

So I think proxy is enabled on all nodes at present.

Wireshark

I got Wireshark set up and am starting to figure out using it. What would be the most useful data for you to see? 

I do not see the yellow and red nodes in the list of devices in Wireshark, which I presume means they are out of range. When I filter to just see data from the node that seems most likely to be the relay for the red node, I do see the occasional "malformed packet". Specifically, it's the CRC that fails. However, this is only seen very occasionally. 

Let me know which buttons to press and I'll get you more data.

Thanks again for your help in figuring this out.

[NickN]

Any further suggestions on this? All comments welcome.