Deployed mesh devices not responding to published messages after Net State Clear with PERSISTENT_STORAGE

[Arif@Lynxemi]

Hi,

We're really stumped on this one. We recently deployed 150 devices running nRF SDK for Mesh. All except three of these devices are running a modified SimpleOnOff Server model, the other three are gateways running the serial example with PERSISTENT_STORAGE turned on. We provisioned the devices using nRF Mesh app for Android which has unicast address of 0x0001 with a range 0x0000 to 0x1999. We used 

Bluetooth Mesh Addr Local Unicast Set

Opcode: 0x9f

to set the three gateways to unicast addresses 0x2004, 0x2006 and 0x2008 respectively. 0x2004 was deployed first, we were able to subscribe and publish messages with it successfully. The other two were installed a couple of days later but weren't tested for a week. Then in the middle of the process of commisioning, we accidentally did a Net State Clear on all three gateways.

We could still see subscription messages sent to group addresses but our 

Bluetooth Mesh Packet Send

Opcode: 0xab

messages were being ignored by the server devices. The server devices could still receive messages from the Android device but not from the gateways. I don't know what I did but about after half a day, gateway 0x2004 started working again, and its published messages were being received and replied to by the server devices. 

The other two are still stuck in limbo. How can we recover from this?

Here's how we publish messages:

1) Read synchronised json file

2) Use 

Bluetooth Mesh Addr Publication Add

Opcode: 0xa4

to get a local publication handle. Remove unused handles if none available.

3) Use Packet Send to publish, received SUCCESS response code. 

Thanks,

Arif

[tesc]

Hi,

I suspect what happened was the sequence number counter was cleared or reset on the devices, which means they started sending out messages with sequence numbers beginning at 0 again. Sequence numbers are sent with all messages, to prevent replay attacks. Receiving devices will discard any message from a sending device, if the sequence number is lower than the highest sequence number seen thus far from that sending device. If this is what happened, then it means the problem will go away once the device has sent as many messages as it had sent at the point when the issue started in the first place.

The way outgoing sequence numbers are treated and stored, power cycling the device will lead to a "jump" in sequence numbers sent of up to around 8 thousand. Power cycling the device will therefore give a jump in sequence number of 8k which could be use to get it "up to speed" quicker. (The reason for this 8k jump is the number is stored to flash in bulks, to reduce flash wear. Large bulks are "reserved" at a time, and the exact number only held in RAM. On reset, the last sequence number sent may be anything within that bulk, so a new bulk is reserved and numbers start from the beginning of that bulk. With the default bulk size of 8k, each power cycle will get the device to start from the beginning of  the next such bulk. This is implementation specific, and not part of the specification. The specification only mandates that sequence numbers are strictly increasing, and this is the way we ensure that with minimal flash writes.)

we accidentally did a Net State Clear on all three gateways.

I am afraid I need some more context to know exactly what we are talking about and what might be the issue, i.e. if my above explanation is what really happened. What SDK (and version) have you built the projects on, and what/how did you do the net state clear on the devices? With answers to those questions I should be able to look into what would have happened.

Regards,
Terje

[Arif@Lynxemi]

Hi tesc, 

Here are the details:

nRF SDK for Mesh v5.0.0 with nrf SDK v17.0.2

s140 7.2.0 softdevice

nrf52840 (mix of Rev 1 and 2)

I suspect you may be right on the money there. Sequence numbers are in the tens of thousands last I probed with IV index of 0. We did net state clear by sending this to the serial module:

Bluetooth Mesh State Clear

Opcode: 0xac

Which resets flash, the DSM and sets seq num and iv index to 0. After that we ran our publish/subscribe add protocol as usual. 

So if I'm reading it right, we have to power cycle our gateway/serial devices until we get a sequence number block greater or equal to current sequence number for that mesh model we're publishing to?

PS I found this post: (+) Join an existing mesh Network with a mesh provisioner (nrf mesh sdk) - Nordic Q&A - Nordic DevZone - Nordic DevZone (nordicsemi.com)

that is sort of similar to our problem except that the author resorts to power cycling the server devices. But we have a lot (150+) server devices to settle if so.

[tesc]

Hi,

Right. Then the main issue is that since the node has sent a message with a high seqnum earlier, other nodes has that high seqnum in their replay protection list, for messages coming from the reset node.

You could also solve this by triggering an IV Update on the network. During IV Update, IV Index increments, and when starting to use the next IV Index the sequence numbers of all nodes start from 0 again. Note however that IV Update is a two-step operation, each step lasting 96 hours, so it would take about four days to get to the working stage. If an IV Update was already ongoing, then the time until everything starts working again would be shorter. Doing it this way, you do not need to reset nodes, but the trade-off is it would take several days before everything is up and running again.

Please note that IV Index for a network should start at a random number, and close to never (due to the huge value span) it would actually be at an actual 0.

Regards,
Terje

[Arif@Lynxemi]

Great, we'll look into a way to implement that and get back to you.

In the meantime, I have this related question/theory:

If, in a large network, I set all the server nodes to publish and subscribe to the same group address. Wouldn't the replay cache for each node fill up such that they inadvertently form mini-meshes within each other and cause what I faced here as well? 

If so, I have to reprovision my whole network to empty the replay cache and set up specific RX and TX groups to get them working properly. Am I right in this understanding?

Regards,

Arif

[tesc]

Hi,

You are correct the replay protection list (RPL) will fill up, and it will reject messages from new senders when the RPL is full.

Discarding old entries in the RPL is a security violation, that opens up for replay attacks.

The RPL size should be large enough for all unicast addresses they will receive messages from, as each sender address has an entry.

If all nodes are broadcasting to all nodes, or worse if all elements on all nodes are broadcasting to one or more elements on all nodes, then yes, the RPL for each node must be large enough to fit all those sender unicast addresses.

Since the message gets discarded if from a sender not in the RPL and the RPL is full, you could get a set of "mini-meshes" as you describe, yes.

It would be a good idea to design your solution such that nodes are not relying on receiving messages from all other nodes in the network.

Regards,
Terje