Deployed mesh devices not responding to published messages after Net State Clear with PERSISTENT_STORAGE

[Arif@Lynxemi]

Hi,

We're really stumped on this one. We recently deployed 150 devices running nRF SDK for Mesh. All except three of these devices are running a modified SimpleOnOff Server model, the other three are gateways running the serial example with PERSISTENT_STORAGE turned on. We provisioned the devices using nRF Mesh app for Android which has unicast address of 0x0001 with a range 0x0000 to 0x1999. We used 

Bluetooth Mesh Addr Local Unicast Set

Opcode: 0x9f

to set the three gateways to unicast addresses 0x2004, 0x2006 and 0x2008 respectively. 0x2004 was deployed first, we were able to subscribe and publish messages with it successfully. The other two were installed a couple of days later but weren't tested for a week. Then in the middle of the process of commisioning, we accidentally did a Net State Clear on all three gateways.

We could still see subscription messages sent to group addresses but our 

Bluetooth Mesh Packet Send

Opcode: 0xab

messages were being ignored by the server devices. The server devices could still receive messages from the Android device but not from the gateways. I don't know what I did but about after half a day, gateway 0x2004 started working again, and its published messages were being received and replied to by the server devices. 

The other two are still stuck in limbo. How can we recover from this?

Here's how we publish messages:

1) Read synchronised json file

2) Use 

Bluetooth Mesh Addr Publication Add

Opcode: 0xa4

to get a local publication handle. Remove unused handles if none available.

3) Use Packet Send to publish, received SUCCESS response code. 

Thanks,

Arif

[tesc]

Hi,

I suspect what happened was the sequence number counter was cleared or reset on the devices, which means they started sending out messages with sequence numbers beginning at 0 again. Sequence numbers are sent with all messages, to prevent replay attacks. Receiving devices will discard any message from a sending device, if the sequence number is lower than the highest sequence number seen thus far from that sending device. If this is what happened, then it means the problem will go away once the device has sent as many messages as it had sent at the point when the issue started in the first place.

The way outgoing sequence numbers are treated and stored, power cycling the device will lead to a "jump" in sequence numbers sent of up to around 8 thousand. Power cycling the device will therefore give a jump in sequence number of 8k which could be use to get it "up to speed" quicker. (The reason for this 8k jump is the number is stored to flash in bulks, to reduce flash wear. Large bulks are "reserved" at a time, and the exact number only held in RAM. On reset, the last sequence number sent may be anything within that bulk, so a new bulk is reserved and numbers start from the beginning of that bulk. With the default bulk size of 8k, each power cycle will get the device to start from the beginning of  the next such bulk. This is implementation specific, and not part of the specification. The specification only mandates that sequence numbers are strictly increasing, and this is the way we ensure that with minimal flash writes.)

we accidentally did a Net State Clear on all three gateways.

I am afraid I need some more context to know exactly what we are talking about and what might be the issue, i.e. if my above explanation is what really happened. What SDK (and version) have you built the projects on, and what/how did you do the net state clear on the devices? With answers to those questions I should be able to look into what would have happened.

Regards,
Terje

[Arif@Lynxemi]

Hi tesc, 

Here are the details:

nRF SDK for Mesh v5.0.0 with nrf SDK v17.0.2

s140 7.2.0 softdevice

nrf52840 (mix of Rev 1 and 2)

I suspect you may be right on the money there. Sequence numbers are in the tens of thousands last I probed with IV index of 0. We did net state clear by sending this to the serial module:

Bluetooth Mesh State Clear

Opcode: 0xac

Which resets flash, the DSM and sets seq num and iv index to 0. After that we ran our publish/subscribe add protocol as usual. 

So if I'm reading it right, we have to power cycle our gateway/serial devices until we get a sequence number block greater or equal to current sequence number for that mesh model we're publishing to?

PS I found this post: (+) Join an existing mesh Network with a mesh provisioner (nrf mesh sdk) - Nordic Q&A - Nordic DevZone - Nordic DevZone (nordicsemi.com)

that is sort of similar to our problem except that the author resorts to power cycling the server devices. But we have a lot (150+) server devices to settle if so.

[tesc]

Hi,

You are correct the replay protection list (RPL) will fill up, and it will reject messages from new senders when the RPL is full.

Discarding old entries in the RPL is a security violation, that opens up for replay attacks.

The RPL size should be large enough for all unicast addresses they will receive messages from, as each sender address has an entry.

If all nodes are broadcasting to all nodes, or worse if all elements on all nodes are broadcasting to one or more elements on all nodes, then yes, the RPL for each node must be large enough to fit all those sender unicast addresses.

Since the message gets discarded if from a sender not in the RPL and the RPL is full, you could get a set of "mini-meshes" as you describe, yes.

It would be a good idea to design your solution such that nodes are not relying on receiving messages from all other nodes in the network.

Regards,
Terje

[Arif@Lynxemi]

Hi Terje,

We tested further and found that the sequence number was not the problem and are leaning to the RPL being the issue. We indeed mistakenly set the groups up that way.

We also noticed we were only receiving subscriptions from 40 devices, which corresponds directly to REPLAY_CACHE_ENTRIES. We have since increased it to 1024, similar to this ticket. This is necessary for our gateways as our network is going to scale up to four thousand devices on one network and we want a minimal number of gateways. 

This may also solve this other ticket you were answering.

So, to architect this mesh I have, I need to set a main TX group which all the nodes publish to and the gateway subscribes but, has less members than the RPL, and a RX group(s) which the gateway publishes to and the nodes subscribe to, which should consist of a small number of gateways.

One more question, although the documentation for the Mesh SDK says that relay messages are not added to the relay nodes' RPL and don't enter the transport layer directly, I cannot find where that distinction happens. In network.c:

uint32_t network_packet_in(const uint8_t * p_packet, uint32_t net_packet_len, const nrf_mesh_rx_metadata_t * p_rx_metadata)
{
    if (p_packet == NULL)
    {
        return NRF_ERROR_NULL;
    }

    const packet_mesh_net_packet_t * p_net_packet = (const packet_mesh_net_packet_t *) p_packet;
    uint32_t status = NRF_SUCCESS;

    /* Create a target buffer to decrypt into, don't have to allocate a new packet. */
    packet_mesh_net_packet_t net_decrypted_packet;

    /* Packet fields not touched by the encryption. */
    memcpy(&net_decrypted_packet,
           p_net_packet,
           net_packet_obfuscation_start_get(p_net_packet) - (uint8_t *) p_net_packet);

    __LOG_XB(LOG_SRC_NETWORK, LOG_LEVEL_DBG1, "  Net RX (enc)", p_packet, net_packet_len);
    network_packet_metadata_t net_metadata;
    status = net_packet_decrypt(&net_metadata,
                                net_packet_len,
                                p_net_packet,
                                &net_decrypted_packet,
                                NET_PACKET_KIND_TRANSPORT);
    if ((status == NRF_SUCCESS) && metadata_is_valid(&net_metadata))
    {
        __LOG_XB(LOG_SRC_NETWORK, LOG_LEVEL_INFO, "Net RX (unenc)", &net_decrypted_packet.pdu[0], net_packet_len);

        uint32_t iv_index = net_metadata.internal.iv_index;
        uint32_t seq_num = net_metadata.internal.sequence_number;
        __LOG(LOG_SRC_NETWORK, LOG_LEVEL_INFO, "RX IV Index: %d\n", iv_index);
        __LOG(LOG_SRC_NETWORK, LOG_LEVEL_INFO, "RX Seq num: %d\n", seq_num);
        NRF_MESH_ASSERT(net_metadata.p_security_material != NULL);

#if MESH_FEATURE_GATT_PROXY_ENABLED
        proxy_net_packet_processed(&net_metadata, p_rx_metadata);
#endif

        const uint8_t * p_net_payload = packet_mesh_net_payload_get(&net_decrypted_packet);

        uint8_t payload_len = net_packet_payload_len_get(&net_metadata, net_packet_len);

        __INTERNAL_EVENT_PUSH(INTERNAL_EVENT_NET_PACKET_RECEIVED, 0, net_packet_len, &net_decrypted_packet);

        status = transport_packet_in((const packet_mesh_trs_packet_t *) p_net_payload,
                                     payload_len,
                                     &net_metadata,
                                     p_rx_metadata);

#if MESH_FEATURE_RELAY_ENABLED
#if MESH_FEATURE_FRIEND_ENABLED
        /* Perform security material translation irrespective whether the received packet was on the
         * friendship security credentials or regular credentials
         */
        if (friend_friendship_established(net_metadata.src))
        {
            nrf_mesh_network_secmat_t * p_tx_secmat = nrf_mesh_net_master_secmat_get(net_metadata.p_security_material);
            if (p_tx_secmat != NULL)
            {
                net_metadata.p_security_material = p_tx_secmat;
            }

        }
#endif
        if (should_relay(&net_metadata, p_rx_metadata)
#if MESH_FEATURE_LPN_ENABLED
            && !mesh_lpn_is_in_friendship()
#endif
            )
        {
            if (packet_relay(&net_metadata, p_net_payload, payload_len, p_rx_metadata) == NRF_SUCCESS)
            {
                msg_cache_entry_add(net_metadata.src, net_metadata.internal.sequence_number);
            }
        }
        else
#endif
        {
            msg_cache_entry_add(net_metadata.src, net_metadata.internal.sequence_number);
        }

    }
    return status;
}

The network packet goes to transport_packet_in before being chosen to relay by should_relay. Does this mean that the message is going to the transport layer and thus being added to the RPL? If so that's a big problem.

Regards,

Arif

[tesc]

Hi,

RPL is at the transport layer, and relay happens at the network layer.

For relay, RPL is not used, but rather the message cache which is the Relay's list of what has recently been relayed (for preventing packets getting sent back-and-forth or in circles.)

From what I understand, the transport layer only adds to RPL any message that is actually addressed to a unicast address on the node, or to a group address to which a model on the node subscribes.

Regards,
Terje

[Arif@Lynxemi]

Hi Terje,

Thank you very much for all your answers, each one of them resolved my queries. We now have a mesh of 151 devices up and running responsively. It was a pain to reprovision them all but it was worth it. Now we can hopefully scale up into the thousands on one network.

Thanks again,

Arif

[tesc]

Hi,

I am happy to hear that things are up and running and scaling well.

I see there is another old thread still pending, are these issues also solved now: Mesh LPN application timing out and/or terminating?

Regards,
Terje

[Arif@Lynxemi]

Hi,

That is still a bit of an issue. I'm going to post another update on that ticket. Those LPN sensors are also going to be integrated into this project we have.

Regards,

Arif

[tesc]

Hi,

Thanks for the update. Let's close this thread then, and continue in the other thread until those issues are solved as well.

Regards,
Terje