Deployed mesh devices not responding to published messages after Net State Clear with PERSISTENT_STORAGE

[Arif@Lynxemi]

Hi,

We're really stumped on this one. We recently deployed 150 devices running nRF SDK for Mesh. All except three of these devices are running a modified SimpleOnOff Server model, the other three are gateways running the serial example with PERSISTENT_STORAGE turned on. We provisioned the devices using nRF Mesh app for Android which has unicast address of 0x0001 with a range 0x0000 to 0x1999. We used 

Bluetooth Mesh Addr Local Unicast Set

Opcode: 0x9f

to set the three gateways to unicast addresses 0x2004, 0x2006 and 0x2008 respectively. 0x2004 was deployed first, we were able to subscribe and publish messages with it successfully. The other two were installed a couple of days later but weren't tested for a week. Then in the middle of the process of commisioning, we accidentally did a Net State Clear on all three gateways.

We could still see subscription messages sent to group addresses but our 

Bluetooth Mesh Packet Send

Opcode: 0xab

messages were being ignored by the server devices. The server devices could still receive messages from the Android device but not from the gateways. I don't know what I did but about after half a day, gateway 0x2004 started working again, and its published messages were being received and replied to by the server devices. 

The other two are still stuck in limbo. How can we recover from this?

Here's how we publish messages:

1) Read synchronised json file

2) Use 

Bluetooth Mesh Addr Publication Add

Opcode: 0xa4

to get a local publication handle. Remove unused handles if none available.

3) Use Packet Send to publish, received SUCCESS response code. 

Thanks,

Arif

[Arif@Lynxemi]

Hi Terje,

We tested further and found that the sequence number was not the problem and are leaning to the RPL being the issue. We indeed mistakenly set the groups up that way.

We also noticed we were only receiving subscriptions from 40 devices, which corresponds directly to REPLAY_CACHE_ENTRIES. We have since increased it to 1024, similar to this ticket. This is necessary for our gateways as our network is going to scale up to four thousand devices on one network and we want a minimal number of gateways. 

This may also solve this other ticket you were answering.

So, to architect this mesh I have, I need to set a main TX group which all the nodes publish to and the gateway subscribes but, has less members than the RPL, and a RX group(s) which the gateway publishes to and the nodes subscribe to, which should consist of a small number of gateways.

One more question, although the documentation for the Mesh SDK says that relay messages are not added to the relay nodes' RPL and don't enter the transport layer directly, I cannot find where that distinction happens. In network.c:

uint32_t network_packet_in(const uint8_t * p_packet, uint32_t net_packet_len, const nrf_mesh_rx_metadata_t * p_rx_metadata)
{
    if (p_packet == NULL)
    {
        return NRF_ERROR_NULL;
    }

    const packet_mesh_net_packet_t * p_net_packet = (const packet_mesh_net_packet_t *) p_packet;
    uint32_t status = NRF_SUCCESS;

    /* Create a target buffer to decrypt into, don't have to allocate a new packet. */
    packet_mesh_net_packet_t net_decrypted_packet;

    /* Packet fields not touched by the encryption. */
    memcpy(&net_decrypted_packet,
           p_net_packet,
           net_packet_obfuscation_start_get(p_net_packet) - (uint8_t *) p_net_packet);

    __LOG_XB(LOG_SRC_NETWORK, LOG_LEVEL_DBG1, "  Net RX (enc)", p_packet, net_packet_len);
    network_packet_metadata_t net_metadata;
    status = net_packet_decrypt(&net_metadata,
                                net_packet_len,
                                p_net_packet,
                                &net_decrypted_packet,
                                NET_PACKET_KIND_TRANSPORT);
    if ((status == NRF_SUCCESS) && metadata_is_valid(&net_metadata))
    {
        __LOG_XB(LOG_SRC_NETWORK, LOG_LEVEL_INFO, "Net RX (unenc)", &net_decrypted_packet.pdu[0], net_packet_len);

        uint32_t iv_index = net_metadata.internal.iv_index;
        uint32_t seq_num = net_metadata.internal.sequence_number;
        __LOG(LOG_SRC_NETWORK, LOG_LEVEL_INFO, "RX IV Index: %d\n", iv_index);
        __LOG(LOG_SRC_NETWORK, LOG_LEVEL_INFO, "RX Seq num: %d\n", seq_num);
        NRF_MESH_ASSERT(net_metadata.p_security_material != NULL);

#if MESH_FEATURE_GATT_PROXY_ENABLED
        proxy_net_packet_processed(&net_metadata, p_rx_metadata);
#endif

        const uint8_t * p_net_payload = packet_mesh_net_payload_get(&net_decrypted_packet);

        uint8_t payload_len = net_packet_payload_len_get(&net_metadata, net_packet_len);

        __INTERNAL_EVENT_PUSH(INTERNAL_EVENT_NET_PACKET_RECEIVED, 0, net_packet_len, &net_decrypted_packet);

        status = transport_packet_in((const packet_mesh_trs_packet_t *) p_net_payload,
                                     payload_len,
                                     &net_metadata,
                                     p_rx_metadata);

#if MESH_FEATURE_RELAY_ENABLED
#if MESH_FEATURE_FRIEND_ENABLED
        /* Perform security material translation irrespective whether the received packet was on the
         * friendship security credentials or regular credentials
         */
        if (friend_friendship_established(net_metadata.src))
        {
            nrf_mesh_network_secmat_t * p_tx_secmat = nrf_mesh_net_master_secmat_get(net_metadata.p_security_material);
            if (p_tx_secmat != NULL)
            {
                net_metadata.p_security_material = p_tx_secmat;
            }

        }
#endif
        if (should_relay(&net_metadata, p_rx_metadata)
#if MESH_FEATURE_LPN_ENABLED
            && !mesh_lpn_is_in_friendship()
#endif
            )
        {
            if (packet_relay(&net_metadata, p_net_payload, payload_len, p_rx_metadata) == NRF_SUCCESS)
            {
                msg_cache_entry_add(net_metadata.src, net_metadata.internal.sequence_number);
            }
        }
        else
#endif
        {
            msg_cache_entry_add(net_metadata.src, net_metadata.internal.sequence_number);
        }

    }
    return status;
}

The network packet goes to transport_packet_in before being chosen to relay by should_relay. Does this mean that the message is going to the transport layer and thus being added to the RPL? If so that's a big problem.

Regards,

Arif

[tesc]

Hi,

RPL is at the transport layer, and relay happens at the network layer.

For relay, RPL is not used, but rather the message cache which is the Relay's list of what has recently been relayed (for preventing packets getting sent back-and-forth or in circles.)

From what I understand, the transport layer only adds to RPL any message that is actually addressed to a unicast address on the node, or to a group address to which a model on the node subscribes.

Regards,
Terje

[Arif@Lynxemi]

Hi Terje,

Thank you very much for all your answers, each one of them resolved my queries. We now have a mesh of 151 devices up and running responsively. It was a pain to reprovision them all but it was worth it. Now we can hopefully scale up into the thousands on one network.

Thanks again,

Arif

[tesc]

Hi,

I am happy to hear that things are up and running and scaling well.

I see there is another old thread still pending, are these issues also solved now: Mesh LPN application timing out and/or terminating?

Regards,
Terje

[Arif@Lynxemi]

Hi,

That is still a bit of an issue. I'm going to post another update on that ticket. Those LPN sensors are also going to be integrated into this project we have.

Regards,

Arif