Recommended migration path and field-update strategy for a Zigbee device moving from NCS v2.4.1 to v3.4.0

[Pawel(embeddedsolutions.pl)]

Product / environment
- SoC: nRF52840
- Current field firmware: nRF Connect SDK v2.4.1 (ZBOSS/Zigbee from that release)
- Target firmware: nRF Connect SDK v3.4.0 with the Zigbee R23 add-on (ncs-zigbee)
- Device role: Zigbee Router (mains-powered), with ZCL attribute reporting configured
- Update mechanism in the field: Zigbee OTA

Background

We are migrating our deployed product from NCS v2.4.1 to v3.4.0, which also bumps the ZBOSS stack (to the R23 add-on). The update will be delivered to field devices over Zigbee FOTA.

While preparing this migration we identified two known issues that appear relevant to our case, both listed in your documentation:

1. KRKNWK-19894 — DFU from NCS v2.5.x or older to v2.6.0+ preserves NVRAM whose `ZB_NVRAM_ZCL_REPORTING_DATA` structure is incompatible with the new firmware, causing corrupted (empty) attribute reporting and, in some cluster configurations, a fatal error on every boot. Our devices come from v2.4.1 and use ZCL reporting, so we expect to hit this.
2. NCSISB-1204 — an interrupted NVRAM write (e.g. power loss during a write) leaves a corrupted record that makes ZBOSS raise a fatal error on boot, resulting in a permanent boot loop / brick.

We understand the documented workaround for KRKNWK-19894 is to erase the NVRAM (fully, or specifically the `ZB_NVRAM_ZCL_REPORTING_DATA` dataset) after the upgrade, at the cost of losing configuration and requiring re-commissioning.

Questions

1. Recommended migration procedure: What is Nordic's recommended way to migrate a fielded Zigbee device from NCS v2.4.1 (old ZBOSS) to v3.4.0 with the R23 add-on, when the update is delivered over Zigbee FOTA? Is there an officially recommended sequence or reference we should follow for this specific stack transition?

2. Mitigating the known issues: Beyond a full/targeted NVRAM erase, is there any recommended way to mitigate KRKNWK-19894 and NCSISB-1204 for our scenario? For NCSISB-1204, the documentation says to request a fixing patch from DevZone — could you confirm whether a fix is already included in the v3.4.0 / R23 ZBOSS binaries, or whether a separate patch is still required?

3. Preserving network membership (avoiding NVRAM erase): Is there a procedure that would allow the device to remain joined to the Zigbee network across this migration — i.e. migrate or convert the existing NVRAM content to the new layout rather than erasing it — so that end users are not forced to re-commission? If any form of NVRAM migration/conversion is feasible, we would prefer that over a wipe.

4. Rollback / downgrade support: Is it possible to support rollback to the previous firmware version (v2.4.1-based) after this migration? Our concern is that once the NVRAM has been written by the new (R23) stack, an older image reading that NVRAM may hit the same layout incompatibility. Is there a supported way to make a downgrade safe?

Note: for safety reasons we cannot confirm the new image in advance. The update must remain revertible, so the new image runs tentatively and can be rolled back if it fails to come up.

Any guidance, reference material, or a fixing patch for the above would be greatly appreciated.

Thank you in advance

[Carl Richard]

Hello Pawel!

Thanks for reaching out with your concerns. We do not have any official migration guide on this, so I've reached out to our developers for some insight on that and your other questions. I do suggest that you read up a bit on our general migration notes, as there may be other significant changes when moving from v2.4.1 to v3.4.0.

I'll come back to you when I know more! 

Best regards,
Carl Richard

[Pawel(embeddedsolutions.pl)]

HI, 

Thanks for fast response.
For greater context, we are already after whole migration related changes.
New firmware boots-up, works fine.

Our concerns are more about the field-update strategy.

Regards,
Pawel

[Carl Richard]

Hello again Pawel!

I've gotten some feedback here now. In order:

1. Recommended migration procedure: What is Nordic's recommended way to migrate a fielded Zigbee device from NCS v2.4.1 (old ZBOSS) to v3.4.0 with the R23 add-on, when the update is delivered over Zigbee FOTA? Is there an officially recommended sequence or reference we should follow for this specific stack transition?

We don't have any official guidelines when it comes to this kind of field migration. I suggest that you, if possible, migrate a one or a few devices initially to see if the behaviour is correct. 

2. Mitigating the known issues: Beyond a full/targeted NVRAM erase, is there any recommended way to mitigate KRKNWK-19894 and NCSISB-1204 for our scenario? For NCSISB-1204, the documentation says to request a fixing patch from DevZone — could you confirm whether a fix is already included in the v3.4.0 / R23 ZBOSS binaries, or whether a separate patch is still required?

For mitigating KRKNWK-19894 it should be enough to follow the workaround described in our Known Issues documentation (under 2.6.1). If you do the targeted erase of ZB_NVRAM_ZCL_REPORTING_DATA it should be possible for the devices to remain in the network as this only wipes the reporting data. As for NCSISB-1204 this is patched in SDK version 2.7.0 (see release notes here), so you should not worry about that in your new firmware. As your devices are mains powered I don't think the risk of NCSISB-1204 creating problems is very big. With that said I can see if I can share that patch with you if you want it. It prevents the bricking of the device, but wipes the NVRAM, requiring recommissioning.
 

3. Preserving network membership (avoiding NVRAM erase): Is there a procedure that would allow the device to remain joined to the Zigbee network across this migration — i.e. migrate or convert the existing NVRAM content to the new layout rather than erasing it — so that end users are not forced to re-commission? If any form of NVRAM migration/conversion is feasible, we would prefer that over a wipe.

As mentioned above you can target only ZB_NVRAM_ZCL_REPORTING_DATA for wiping. Then the devices should remain joined to the network. 

4. Rollback / downgrade support: Is it possible to support rollback to the previous firmware version (v2.4.1-based) after this migration? Our concern is that once the NVRAM has been written by the new (R23) stack, an older image reading that NVRAM may hit the same layout incompatibility. Is there a supported way to make a downgrade safe?

After NVRAM is wiped it's, as far as I understand, not possible to rollback, but using MCUBoot (with test swap) it should be possible to test the new image partially. This will likely trigger KRKNWK-19894, but crashing/bootloop should be handled by MCUBoot, reverting the application to the old firmware. In this configuration the NVRAM (or ZB_NVRAM_ZCL_REPORTING_DATA) must be wiped from the old firmware or from the new firmware (at some point before an eventual issue with KRKNWK-19894) when the new firmware is confirmed.

Are you able to do some bench testing on a fielded device to test out the migration?

Best regards,
Carl Richard