access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Wed, 10 Jun 2020 14:12:46 GMT

Thank you for the suggestion, but as mentioned previously we:

- decreased NETWORK_SEQNUM_FLASH_BLOCK_SIZE

- fixed the underlying low-power resetting issue

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

Igor — Fri, 05 Jun 2020 13:22:38 GMT

Hung Bui What about NETWORK_MAX_IV_UPDATE_INTERVAL_MINUTES ? Should it be changed from 144*60 to 6*60?

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Mon, 11 May 2020 18:14:34 GMT

Thank you for the info.

For now we will go with the given workaround.

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

Hung Bui — Thu, 07 May 2020 08:20:10 GMT

Hi Sirio,
What you do can remedy the issue but won't solve it. One of our engineer suggest a list of thing you can do to remedy as follow:

Workaround

Updating the following parameter

the minimum time requirement for triggering IV update from 96 hours to 4 hours
- Modify IV_UPDATE_TIMEOUT_MINUTES
The minimum gap between each IV update from 192 hours to 8 hours
- Modify IV_RECOVERY_TIMEOUT_MINUTES
Decrease the SEQ block number NETWORK_SEQNUM_FLASH_BLOCK_SIZE from 8192 to 512
Add fail-safe protection: If SEQ is going to reach maximum number but IV update cannot be triggered, then unprovisioned itself

Risk

It is more easy to trigger the IV update and increasing IV
But if we do IV update every day, the product still can run 11,767,033 years
- 2 ^ 32 / 365 = 11767033 years
This issue has been fixed in Mesh SDK v4.0, the stack will write the “iv_update_timer” into flash every 30 mins (IV_UPDATE_TIMEOUT_PERIODIC_SAVE_MINUTES)

This won't fix the issue as the best solution in my opinion is to be able to recover the sequence number after the battery died instead of unprovisioning. I will continue to check with the team if that possible.

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Wed, 06 May 2020 15:37:55 GMT

Thank you very much for getting back to me.

The problem seems to have been that these devices are on battery power and, as they began running low, were self-resetting when voltage would drop while turning on the radio to transmit.

Then, after reset they would increment the flash block.

After about 2k cycles of this they would reach the maximum sequence number.

Then someone would notice they were not working and would replace their battery, but by this time they would be out of sequence numbers and no longer able to send any new messages to the network.

We are debugging the power consumption (and correct power-sensing on boot) separately, but to avoid this issue have also added this line to nrf_mesh_config_app.h:

#define NETWORK_SEQNUM_FLASH_BLOCK_SIZE 512

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

Hung Bui — Wed, 06 May 2020 08:08:12 GMT

Hi Sirio,

I'm sorry I was dragged into another project.
I didn't know that your device reboot frequently. Could you describe the behaviour of your LPN node ? How often it reboot, what make it wake up ? What kind of sleep did you put it in ?

The LPN node is not supposed to reboot, it supposed to sleep (System ON CPU Idle) not rebooting on every waking up. The reason why the sequence number jumping was that we don't store the sequence number into flash after each increment instead we store the block it's in. The next reboot we will jump to the next block. Rebooting will cause the sequence number jumping from block to block.

Regarding the friend node, we don't provide a friend node project in SDK v3.2 only a hex file, did you use that hex file ?

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Tue, 05 May 2020 12:58:42 GMT

Please, I am onsite trying to fix these devices that are deployed in production.

Any help you ca give urgently is much appreciated.

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Mon, 04 May 2020 18:56:16 GMT

Seems I have found the source of the error: on device reboot, 'seqnum' is being set to the previously saved 'seqnum_max_available' and 'seqnum_max_available' makes a large jump.

See log:

<t:          0>, main.c, 1051, ----- LPN Remote -----
<t:      20049>, mesh_app_utils.c,   65, Device UUID (raw): 6F2F379A0492F34C85E49D124B747639
<t:      20054>, mesh_app_utils.c,   70, Device UUID : 9A372F6F-9204-4CF3-85E4-9D124B747639
<t:    8169629>, ble_softdevice_support.c,  104, Successfully updated connection parameters
<t:    8679374>, main.c,  771, NRF_MESH_EVT_IV_UPDATE_NOTIFICATION
<t:    8702932>, main.c,  735, NRF_MESH_EVT_DISABLED
<t:    8703515>, main.c,  859, Successfully provisioned
<t:    8703521>, main.c,  873, Node Address: 0x36F2
<t:    8810658>, ble_softdevice_support.c,  104, Successfully updated connection parameters
<t:    9116799>, net_state.c,  561, seqnum: 0; seqnum_max_available 8192; iv_index 9
<t:    9132528>, net_state.c,  561, seqnum: 1; seqnum_max_available 8192; iv_index 9
<t:    9203315>, net_state.c,  561, seqnum: 2; seqnum_max_available 8192; iv_index 9
<t:    9203336>, net_state.c,  561, seqnum: 3; seqnum_max_available 8192; iv_index 9
<t:    9203347>, net_state.c,  561, seqnum: 4; seqnum_max_available 8192; iv_index 9
<t:    9203357>, net_state.c,  561, seqnum: 5; seqnum_max_available 8192; iv_index 9
<t:    9203367>, net_state.c,  561, seqnum: 6; seqnum_max_available 8192; iv_index 9
<t:    9203384>, net_state.c,  561, seqnum: 7; seqnum_max_available 8192; iv_index 9
<t:    9203397>, net_state.c,  561, seqnum: 8; seqnum_max_available 8192; iv_index 9
<t:    9203411>, net_state.c,  561, seqnum: 9; seqnum_max_available 8192; iv_index 9
<t:    9462835>, net_state.c,  561, seqnum: 10; seqnum_max_available 8192; iv_index 9
<t:    9722376>, net_state.c,  561, seqnum: 11; seqnum_max_available 8192; iv_index 9
<t:    9722397>, config_server.c,  629, dsm_appkey_add(appkey_handle:0 appkey_index:0)
<t:    9722402>, net_state.c,  561, seqnum: 12; seqnum_max_available 8192; iv_index 9
<t:   11358151>, config_server.c, 2451, Access  Info:
		element_index=0		model_id = 2-788		model_handle=27
<t:   11358159>, net_state.c,  561, seqnum: 13; seqnum_max_available 8192; iv_index 9
<t:   11625540>, config_server.c, 2451, Access  Info:
		element_index=1		model_id = 1001-FFFF		model_handle=4
<t:   11625548>, net_state.c,  561, seqnum: 14; seqnum_max_available 8192; iv_index 9
<t:   11885064>, config_server.c, 2451, Access  Info:
		element_index=2		model_id = 1311-FFFF		model_handle=10
<t:   11885072>, net_state.c,  561, seqnum: 15; seqnum_max_available 8192; iv_index 9
<t:   12144588>, config_server.c, 2451, Access  Info:
		element_index=3		model_id = 1311-FFFF		model_handle=14
<t:   12144596>, net_state.c,  561, seqnum: 16; seqnum_max_available 8192; iv_index 9
<t:   12396249>, config_server.c, 2451, Access  Info:
		element_index=4		model_id = 1001-FFFF		model_handle=16
<t:   12396257>, net_state.c,  561, seqnum: 17; seqnum_max_available 8192; iv_index 9
<t:   12655786>, net_state.c,  561, seqnum: 18; seqnum_max_available 8192; iv_index 9
<t:   12655804>, net_state.c,  561, seqnum: 19; seqnum_max_available 8192; iv_index 9
<t:   12655818>, net_state.c,  561, seqnum: 20; seqnum_max_available 8192; iv_index 9
<t:   12907446>, net_state.c,  561, seqnum: 21; seqnum_max_available 8192; iv_index 9
<t:   12907464>, net_state.c,  561, seqnum: 22; seqnum_max_available 8192; iv_index 9
<t:   12907477>, net_state.c,  561, seqnum: 23; seqnum_max_available 8192; iv_index 9
<t:   13166970>, net_state.c,  561, seqnum: 24; seqnum_max_available 8192; iv_index 9
<t:   13166988>, net_state.c,  561, seqnum: 25; seqnum_max_available 8192; iv_index 9
<t:   13167001>, net_state.c,  561, seqnum: 26; seqnum_max_available 8192; iv_index 9
<t:   13418630>, net_state.c,  561, seqnum: 27; seqnum_max_available 8192; iv_index 9
<t:   13418648>, net_state.c,  561, seqnum: 28; seqnum_max_available 8192; iv_index 9
<t:   13418661>, net_state.c,  561, seqnum: 29; seqnum_max_available 8192; iv_index 9
<t:   13678146>, net_state.c,  561, seqnum: 30; seqnum_max_available 8192; iv_index 9
<t:   13693877>, net_state.c,  561, seqnum: 31; seqnum_max_available 8192; iv_index 9
<t:   13709606>, net_state.c,  561, seqnum: 32; seqnum_max_available 8192; iv_index 9
<t:   13725335>, net_state.c,  561, seqnum: 33; seqnum_max_available 8192; iv_index 9
<t:   13741064>, net_state.c,  561, seqnum: 34; seqnum_max_available 8192; iv_index 9
<t:   13756793>, net_state.c,  561, seqnum: 35; seqnum_max_available 8192; iv_index 9
<t:   13772522>, net_state.c,  561, seqnum: 36; seqnum_max_available 8192; iv_index 9
<t:   13788251>, net_state.c,  561, seqnum: 37; seqnum_max_available 8192; iv_index 9
<t:   13866887>, net_state.c,  561, seqnum: 38; seqnum_max_available 8192; iv_index 9

##
# device rebooted
##

<t:          0>, main.c, 1051, ----- LPN Remote -----
<t:      15363>, mesh_app_utils.c,   65, Device UUID (raw): 6F2F379A0492F34C85E49D124B747639
<t:      15368>, mesh_app_utils.c,   70, Device UUID : 9A372F6F-9204-4CF3-85E4-9D124B747639
<t:   15748373>, net_state.c,  561, seqnum: 8192; seqnum_max_available 16384; iv_index 9
<t:   15752238>, main.c,  672, Friendship: received offer from 0x6AB6
<t:   15752253>, net_state.c,  561, seqnum: 8193; seqnum_max_available 16384; iv_index 9
<t:   15755604>, main.c,  693, Friendship: established with 0x6AB6
<t:   15755612>, net_state.c,  561, seqnum: 8194; seqnum_max_available 16384; iv_index 9
<t:   16086715>, net_state.c,  561, seqnum: 8195; seqnum_max_available 16384; iv_index 9
<t:   16090182>, net_state.c,  561, seqnum: 8196; seqnum_max_available 16384; iv_index 9
<t:   16421262>, net_state.c,  561, seqnum: 8197; seqnum_max_available 16384; iv_index 9
<t:   16424729>, net_state.c,  561, seqnum: 8198; seqnum_max_available 16384; iv_index 9
<t:   16428195>, net_state.c,  561, seqnum: 8199; seqnum_max_available 16384; iv_index 9
<t:   16759275>, net_state.c,  561, seqnum: 8200; seqnum_max_available 16384; iv_index 9
<t:   16762741>, net_state.c,  561, seqnum: 8201; seqnum_max_available 16384; iv_index 9
<t:   16766207>, net_state.c,  561, seqnum: 8202; seqnum_max_available 16384; iv_index 9
<t:   16769673>, net_state.c,  561, seqnum: 8203; seqnum_max_available 16384; iv_index 9
<t:   16773139>, net_state.c,  561, seqnum: 8204; seqnum_max_available 16384; iv_index 9
<t:   16776605>, net_state.c,  561, seqnum: 8205; seqnum_max_available 16384; iv_index 9
<t:       2855>, main.c,  718, Friendship: terminated with 0x6AB6.
	Reason: NRF_MESH_EVT_FRIENDSHIP_TERMINATED_REASON_NO_REPLY

##
# device rebooted
##

<t:          0>, main.c, 1051, ----- LPN Remote -----
<t:       9310>, mesh_app_utils.c,   65, Device UUID (raw): 6F2F379A0492F34C85E49D124B747639
<t:       9314>, mesh_app_utils.c,   70, Device UUID : 9A372F6F-9204-4CF3-85E4-9D124B747639
<t:     104547>, net_state.c,  561, seqnum: 16384; seqnum_max_available 24576; iv_index 9
<t:     104561>, client.c,  243, Sending msg: ONOFF SET 1
<t:     104566>, net_state.c,  561, seqnum: 16385; seqnum_max_available 24576; iv_index 9
<t:     104580>, net_state.c,  561, seqnum: 16386; seqnum_max_available 24576; iv_index 9
<t:     104593>, net_state.c,  561, seqnum: 16387; seqnum_max_available 24576; iv_index 9
<t:     107882>, net_state.c,  561, seqnum: 16388; seqnum_max_available 24576; iv_index 9
<t:     110055>, main.c,  672, Friendship: received offer from 0x6B68
<t:     110069>, net_state.c,  561, seqnum: 16389; seqnum_max_available 24576; iv_index 9
<t:     111158>, net_state.c,  561, seqnum: 16390; seqnum_max_available 24576; iv_index 9
<t:     113417>, main.c,  555, Fstate: fast-polling every 10 sec
<t:     113420>, main.c,  693, Friendship: established with 0x6B68
<t:     114436>, net_state.c,  561, seqnum: 16391; seqnum_max_available 24576; iv_index 9

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Mon, 04 May 2020 17:07:55 GMT

I added a print to get the IV index, I'm seeing (on a different device with the same error):

<t:     153301>, net_state.c,  592, seqnum: 16777216; seqnum_max_available 16777216; iv_index 8
<t:     153306>, main.c,  456, initiate friendship
<t:     153309>, client.c,  243, Sending msg: ONOFF SET 0
<t:     153314>, net_state.c,  592, seqnum: 16777216; seqnum_max_available 16777216; iv_index 8
<t:     153319>, client.c,  221, NRF_ERROR_FORBIDDEN
<t:     189352>, net_state.c,  592, seqnum: 16777216; seqnum_max_available 16777216; iv_index 8
<t:     225403>, net_state.c,  592, seqnum: 16777216; seqnum_max_available 16777216; iv_index 8
<t:     261454>, net_state.c,  592, seqnum: 16777216; seqnum_max_available 16777216; iv_index 8
<t:     297505>, net_state.c,  592, seqnum: 16777216; seqnum_max_available 16777216; iv_index 8
<t:     333556>, net_state.c,  592, seqnum: 16777216; seqnum_max_available 16777216; iv_index 8
<t:     369606>, main.c,  710, Friendship: request timed out

The network is currently on IV 9

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Mon, 04 May 2020 16:26:35 GMT

Yes, looks like 16777216 is NETWORK_SEQNUM_MAX+1.

These LPN nodes are part of a large network with about 1600 nodes.

All friends are running Mesh SDK v3.2.0.

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

Hung Bui — Mon, 04 May 2020 15:31:35 GMT

Seems that the sequence number reached the maximum without IV index update.
Could you give information about the friend node you use ? Which firmware on it ? Is it Nordic's device ? Which SDK ?

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Mon, 04 May 2020 14:59:18 GMT

I got access to one of the devices on site.

This is what we're seeing with that debug print:

<t:     140704>, net_state.c,  592, m_net_state.seqnum: 16777216; seqnum_max_available 16777216
<t:     140709>, main.c,  456, initiate friendship
<t:     140713>, client.c,  243, Sending msg: ONOFF SET 0
<t:     140717>, net_state.c,  592, m_net_state.seqnum: 16777216; seqnum_max_available 16777216
<t:     140723>, client.c,  221, NRF_ERROR_FORBIDDEN
<t:     176755>, net_state.c,  592, m_net_state.seqnum: 16777216; seqnum_max_available 16777216
<t:     212807>, net_state.c,  592, m_net_state.seqnum: 16777216; seqnum_max_available 16777216
<t:     248859>, net_state.c,  592, m_net_state.seqnum: 16777216; seqnum_max_available 16777216
<t:     284911>, net_state.c,  592, m_net_state.seqnum: 16777216; seqnum_max_available 16777216
<t:     320963>, net_state.c,  592, m_net_state.seqnum: 16777216; seqnum_max_available 16777216
<t:     357014>, main.c,  710, Friendship: request timed out

This seems to be

1x seqnum error trying to send ONOFF SET 0

5x seqnum errors trying to establish friendship

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Wed, 29 Apr 2020 12:11:14 GMT

Hi Hung,

Thank you very much.

Next time I have physical access to the site I will be able to push a --sectorerase and get you that debug print.

No success reproducing in the lab - have you been able to track down on your side how this could be happening?

The devices are already on v3.2.0 so if we can find a fix for v3.2.0 then we would be able to --sectorerase the devices instead of re-flashing and re-provisioning and re-programming.

For reference, these are the full details of the build:

-- Configuring CMake for nRF5 SDK for Bluetooth Mesh 3.2.0
-- SDK: nRF5_SDK_15.3.0_59ac345
-- Platform: nrf52832_xxAA
-- Arch: cortex-m4f
-- SoftDevice: s132_6.1.1
-- Board: pca10040
-- The C compiler identification is GNU 8.3.1

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

Hung Bui — Wed, 29 Apr 2020 08:46:25 GMT

Hi Sirio,

We would need to get the log from the defected product to see what exact the issue was. Could you try printing out the sequence number as suggested earlier ?
Have you managed to reproduce the issue in the lab ? Maybe you need to push it to send data quicker.
According to the migration guide here, even moving from v3.1 to v3.2 you would need to do reprovisioning. So maybe it's better to find the solution to fix it on v3.1
Could you give more information about the friend node you used ?

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Tue, 28 Apr 2020 19:13:10 GMT

Hello :)

So, looks like moving this app to SDK v4.1.0 is going to entail flashing, re-provisioning and re-configuring several *hundred* devices already deployed in the field.

If a fix can be found with v3.2.0 then the devices can be flashed with --sectorerase and maintain their provisioning data and config.

Please advise.

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Fri, 24 Apr 2020 22:21:22 GMT

FYI moving to SDK v4.1.0 is proving challenging; see this issue that came up as a result: devzone.nordicsemi.com/.../sdk-for-meshv-v4-1-0-access_model_publish-triggers-an-assert-on-bearer_event_in_correct_irq_priority-but-same-code-is-in-prod-on-sdk-for-meshv-v3-2-0

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Tue, 21 Apr 2020 13:13:04 GMT

Thank you very much.

We will investigate moving to SDK v4.10.

In the meantime, the relevant defines for LPN are:

#define MESH_FEATURE_LPN_ENABLED 1

#define POLL_TIMEOUT_MS (MIN_TO_MS(24)) /**< Upper limit for two subsequent Friend Polls */

#define RECEIVE_DELAY_MS (100) /**< Time between LPN sending and listening for a response */

#define FRIEND_REQUEST_TIMEOUT_MS \
    (MESH_LPN_FRIEND_REQUEST_TIMEOUT_MAX_MS) /**< Max time to scan for incoming Friend Offers */

On the friend nodes, app_config.h looks like:

/* all other parameters are left to SDK default */
#define MESH_FEATURE_FRIEND_ENABLED 1

Next time I have one these devices on hand (they are installed on-site) I will push an update with the suggested debug print of `seqnum`.

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

Hung Bui — Tue, 21 Apr 2020 11:39:45 GMT

Hi Sirio,

We had some issues with the sequence number and the IV update (related to the timer to trigger IV Update to avoid the sequence number to overrun)
They are resolved in our newer SDK (from v4.0). So our suggestion is to migrate to the new SDK to make sure the issue is solved.

If it's not possible to move to SDK v4.x, we may need to debug and find the root cause on your device.

I would suggest to print a log of the sequence number (m_net_state.seqnum ) at this location: https://github.com/NordicSemiconductor/nRF5-SDK-for-Mesh/blob/08c60f6d8766bdb83bb108ec3fbd24d758689235/mesh/core/src/net_state.c#L584

We also need to check the friend nodes if they was operating normally when the issue occurs. How many friend nodes were around the defected LPN node ? Can these friend nodes establish connection with a normal node when the issue happens ?
Which Friend node firmware was running ? Please provide more information about the friendship configuration (POLL_TIMEOUT_MS, FRIEND_REQUEST_TIMEOUT_MS, etc)

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

sirio — Fri, 17 Apr 2020 12:45:19 GMT

Thank you for looking into this :)

I believe we are sleeping, not turning off - the main loop looks like:

    for (;;)
    {
        sd_app_evt_wait();
    }

General operation:

- woken from sleep by a GPIOTE event on a button press

- check friendship status, if not in friendship then initiate new friendship

- send UNACK messages from various client models

- hit sd_app_evt_wait() and go back to sleep (about 3.5µa power consumption)

RE: access_model_publish() returns NRF_ERROR_FORBIDDEN in LPN and all friendships time out

Hung Bui — Thu, 16 Apr 2020 13:33:10 GMT

Hi Sirio,

I have forwarded the issue to our mesh team and is waiting for their feedback.

Could you give some more information about how the LPN node work ?

When it enters sleep mode, would it go to System Off mode or it would enter System ON Idle (only sleep no reset) ?
NRF_ERROR_FORBIDDEN from access_model_publish() telling that it could be an issue with the sequence number.