Issues with completing DFU after updated to NCS V2.6.0

RE: Issues with completing DFU after updated to NCS V2.6.0

Mike Austin (LPI) — Thu, 30 May 2024 20:54:56 GMT

Thanks Vidar.

We have something working now - "hacked" the smt_bt.c file so that with CONFIG_MCUMGR_TRANSPORT_BT_AUTHEN=n, we set the permissions on the SMP service to BT_GATT_PERM_WRITE_ENCRYPT & BT_GATT_PERM_READ_ENCRYPT. If its set to "y", then it defaults to BT_GATT_PERM_WRITE_AUTHEN & BT_GATT_PERM_READ_AUTHEN.

We're still not sure how we managed to get the upgrade from v1.6 to v1.7 of our firmware to work, as it looks like we had set CONFIG_MCUMGR_SMP_BT_AUTHEN=y in v1.6. For the product we have out in the field, we're trying another "hack" in our App that detects a device that has firmware v1.6 and approaches the OTA process slightly differently.

Appreciate all your help in getting a resolution to this one! Certainly one of the more challenging tickets I've had up on DevZone

Regards,

Mike

RE: Issues with completing DFU after updated to NCS V2.6.0

Vidar Berg — Wed, 29 May 2024 12:21:52 GMT

Hi Mike,

The 'erase settings' option will cause the bonding information to be deleted, so I think this may explain some of the problems you've been experiencing, at least.

CONFIG_MCUMGR_SMP_BT_AUTHEN was renamed to CONFIG_MCUMGR_TRANSPORT_BT_AUTHEN in Zephyr v.3.3 (https://github.com/nrfconnect/sdk-zephyr/blob/main/doc/releases/release-notes-3.3.rst), and is used in the service declaration here: https://github.com/nrfconnect/sdk-zephyr/blob/db34adf9c3d856b15f9662e034e44727aa16de47/subsys/mgmt/mcumgr/transport/src/smp_bt.c#L355

[quote user="Mike Austin (LPI)"]Interestingly, in our factory code, we have this set to "n", but in our operational code (in both v1.6 and v1.7) we have this set to "y", yet the upgrade from v1.6 to v1.7 [/quote]

That is strange. Did you verify that the symbol ended up being selected in the generated .config file?

[quote user="Mike Austin (LPI)"]

As we're sending an unencrypted file, it seems we need the encrypted connection but as we don't have any means of entering or displaying passkeys, we can't have the requirement for authentication.

Is there a way to have BT_GATT_PERM_WRITE_ENCRYPT & BT_GATT_PERM_READ_ENCRYPT permission setting on the SMP service within v2.2.0 and v2.6.0 without needing to "hack" the Zephyr files?

[/quote]

Authentication limits who can initiate DFU (typcically those who have physical access to the device), but I don't think encryption itself offers much added security as the update binary is not encrypted in the app.

You can change the sec. level to require encryption, but this requires the smp service to be modified as there is no Kconfig symbol setting to only enable encryption.

https://github.com/nrfconnect/sdk-zephyr/blob/db34adf9c3d856b15f9662e034e44727aa16de47/subsys/mgmt/mcumgr/transport/src/smp_bt.c#L355

Regards,

Vidar

RE: Issues with completing DFU after updated to NCS V2.6.0

Mike Austin (LPI) — Wed, 29 May 2024 03:48:17 GMT

Hi Vidar,

We tried turning that setting off, but it had on effect on our ability to do a DFU OTA.

We did some logging from within the nRF Connect App to see if we could get some further information. We currently have 3 versions of firmware we are using for testing:

V1.6 - this is what is currently in field deployed devices, and is built on NCS v2.2.0

V1.7 - this is an upgrade that we were planning to release, but when we tested it noticed it had a few bugs we hadn't uncovered in V1.6. This is also built on NCS V2.2.0

V1.8 - this is the newest version, that fixes the bugs in V1.7, but it built around NCS V2.6.0

We tried going from V1.6 -> V1.7 -> V1.8. In each case we had nRF Connect set to "Test and Confirm",

Upgrade from V1.6 (NCS V2.2.0) to V1.7 (NCS V2.2.0)

Found valid Firmware in file:///private/var/mobile/Containers/Data/Application/C16C782E-248A-43B7-98F8-B64C510541BD/Documents/6_update_1.7.bin for Device DFU McuMgr.

Upgrade started with 1 image(s) using 'Test and Confirm' mode

Firmware Upgrade Started.

State changed from none to requestMcuMgrParameters

Peripheral connected

Device ready

Mcu Manager parameters received (4 x 2475)

State changed from requestMcuMgrParameters to bootloaderInfo

Bootloader info not supported

State changed from bootloaderInfo to validate

Image List response: Header: {"version": "0", "op": "1", "flags": 0, "length": 134, "group": 1, "seqNum": 130, "commandId": 0}, Payload: {"images" : {{"bootable" : true, "version" : "0.0.0", "hash" : 0xC25B88AA20BEC2CC7E5BE0415A93344EF27AC17E641194893AAC3D88BFDE0A45, "pending" : false, "active" : true, "permanent" : false, "confirmed" : true, "slot" : 0}}, "splitStatus" : 0}

Scheduling upload (hash: 0x2A78A535D1BAB0A61ED01DEC2872F2CB60C6B15DE7A26AF0C2E3DD2212FB4079) for image 0 (slot: 1)

State changed from validate to upload

No remaining chunks to be sent? chunkOffset: 203620, imageData: 203620.

No remaining chunks to be sent? chunkOffset: 203620, imageData: 203620.

Upload finished (1 of 1)

State changed from upload to test

Image Test response: Header: {"version": "0", "op": "3", "flags": 0, "length": 244, "group": 1, "seqNum": 215, "commandId": 0}, Payload: {"images" : {{"hash" : 0xC25B88AA20BEC2CC7E5BE0415A93344EF27AC17E641194893AAC3D88BFDE0A45, "confirmed" : true, "active" : true, "slot" : 0, "pending" : false, "version" : "0.0.0", "permanent" : false, "bootable" : true}, {"confirmed" : false, "version" : "0.0.0", "bootable" : true, "hash" : 0x2A78A535D1BAB0A61ED01DEC2872F2CB60C6B15DE7A26AF0C2E3DD2212FB4079, "pending" : true, "slot" : 1, "active" : false, "permanent" : false}}, "splitStatus" : 0}

State changed from test to reset

Reset request confirmed

Disconnected.

Peripheral disconnected

Device has disconnected

Waiting 10 seconds reconnecting...

Reconnecting...

Reconnect deferred

State changed from reset to confirm

Peripheral connected

Device ready

Image Confirm response: Header: {"version": "0", "op": "3", "flags": 0, "length": 244, "group": 1, "seqNum": 216, "commandId": 0}, Payload: {"splitStatus" : 0, "images" : {{"hash" : 0x2A78A535D1BAB0A61ED01DEC2872F2CB60C6B15DE7A26AF0C2E3DD2212FB4079, "bootable" : true, "pending" : false, "permanent" : false, "active" : true, "version" : "0.0.0", "slot" : 0, "confirmed" : true}, {"hash" : 0xC25B88AA20BEC2CC7E5BE0415A93344EF27AC17E641194893AAC3D88BFDE0A45, "bootable" : true, "permanent" : false, "slot" : 1, "confirmed" : false, "pending" : false, "version" : "0.0.0", "active" : false}}}

Upgrade complete

State changed from confirm to success

Success!

Upgrade from V1.7 (NCS V2.2.0) to V1.8 (NCS V2.6.0)

Upgrade started with 1 image(s) using 'Test and Confirm' mode

Firmware Upgrade Started.

State changed from none to requestMcuMgrParameters

Peripheral connected

Device ready

Mcu Manager parameters received (4 x 2475)

State changed from requestMcuMgrParameters to bootloaderInfo

Bootloader info not supported

State changed from bootloaderInfo to validate

Image List response: Header: {"version": "0", "op": "1", "flags": 0, "length": 244, "group": 1, "seqNum": 116, "commandId": 0}, Payload: {"images" : {{"permanent" : false, "confirmed" : true, "bootable" : true, "pending" : false, "hash" : 0x2A78A535D1BAB0A61ED01DEC2872F2CB60C6B15DE7A26AF0C2E3DD2212FB4079, "version" : "0.0.0", "active" : true, "slot" : 0}, {"pending" : false, "slot" : 1, "bootable" : true, "version" : "0.0.0", "active" : false, "permanent" : false, "confirmed" : false, "hash" : 0xC25B88AA20BEC2CC7E5BE0415A93344EF27AC17E641194893AAC3D88BFDE0A45}}, "splitStatus" : 0}

Secondary slot of image 0 will be overwritten

Scheduling upload (hash: 0x79834FF4F72A0934C370B232E754CD1FF56971FCCB565A7593A4B3A88EE487CA) for image 0 (slot: 1)

State changed from validate to upload

Retry 1 for seq: 159

Retry 1 for seq: 181

No remaining chunks to be sent? chunkOffset: 207260, imageData: 207260.

No remaining chunks to be sent? chunkOffset: 207260, imageData: 207260.

Upload finished (1 of 1)

State changed from upload to test

Image Test response: Header: {"version": "0", "op": "3", "flags": 0, "length": 244, "group": 1, "seqNum": 208, "commandId": 0}, Payload: {"splitStatus" : 0, "images" : {{"pending" : false, "version" : "0.0.0", "confirmed" : true, "hash" : 0x2A78A535D1BAB0A61ED01DEC2872F2CB60C6B15DE7A26AF0C2E3DD2212FB4079, "slot" : 0, "permanent" : false, "active" : true, "bootable" : true}, {"confirmed" : false, "hash" : 0x79834FF4F72A0934C370B232E754CD1FF56971FCCB565A7593A4B3A88EE487CA, "bootable" : true, "slot" : 1, "active" : false, "version" : "0.0.0", "pending" : true, "permanent" : false}}}

State changed from test to reset

Reset request confirmed

Peripheral disconnected

Device has disconnected

Waiting 10 seconds reconnecting...

Disconnected.

Reconnecting...

Reconnect deferred

State changed from reset to confirm

Peripheral connected

The handle is invalid.

Request (SMPv1, group: image, seq: 209, command: state) failed: The handle is invalid.

The handle is invalid.

DFU failed: The handle is invalid.

DFU Failed with Error: The handle is invalid.

A colleague stumbled across this ticket, which seems to indicate that with CONFIG_MCUMGR_SMP_BT_AUTHEN=y, that you need encryption and authentication. Interestingly, in our factory code, we have this set to "n", but in our operational code (in both v1.6 and v1.7) we have this set to "y", yet the upgrade from v1.6 to v1.7 works as long as the device is paired, which sort of doesn't make sense

(Note - we don't have any display options for passkeys on our device, so can't authenticate)

In NCS v2.6.0, the config setting CONFIG_MCUMGR_SMP_BT_AUTHEN doesn't seem to be valid and I can't find anywhere in the corresponding smt_bt.c file where the SMP service is declared, so I can't see how its configured in terms of encryption/authentication

As we're sending an unencrypted file, it seems we need the encrypted connection but as we don't have any means of entering or displaying passkeys, we can't have the requirement for authentication.

Is there a way to have BT_GATT_PERM_WRITE_ENCRYPT & BT_GATT_PERM_READ_ENCRYPT permission setting on the SMP service within v2.2.0 and v2.6.0 without needing to "hack" the Zephyr files?

Regards,

Mike

RE: Issues with completing DFU after updated to NCS V2.6.0

Vidar Berg — Fri, 24 May 2024 12:17:17 GMT

Sorry for not noticing this earlier, but this setting will erase the settings partition when you do DFU:

Do you experience the same if you turn it off? I have requested that we make this disabled by default

RE: Issues with completing DFU after updated to NCS V2.6.0

Vidar Berg — Thu, 23 May 2024 11:28:37 GMT

Hi Mike,

Is it possible to enable logging in the new version to see if there are any errors reported? E.g., if the link fails to be secured. You can also use the bt_foreach_bond() function to find out how many bonds are stored to check that bonds are not being erased during the DFU process.

Regards,

Vidar

RE: Issues with completing DFU after updated to NCS V2.6.0

Mike Austin (LPI) — Thu, 23 May 2024 05:22:40 GMT

Hi Vidar,

OK, checked the Attributes Table order between the two versions of firmware built around the two versions of NCS, and they are exactly the same.

So, something seems to have changed in how the SMP service is set up.

To try and get around this in the short term (we have product that customers are unable to update the firmware for, so I'm trying to get something that will work), I did the following:

Went back to NCS v2.2.0/Zephyr v3.2.99
Took the version of firmware that used to work, which was v1.7. This successfully upgrades from v1.6 via our custom app.
Made the very minor changes that I need to make to sort out the bug we have uncovered
Set that as v1.9 (v1.8 is the one built around NCS v2.6.0 that we have been having issues with) and did a pristine build, using NCS v2.2.0
Attempted to do the upgrade via the Device Manager

Testing using Device Manager, we are now seeing something similar to before, even though we aren't changing the version of NCS/Zephyr now! The steps are the same, and the failure occurs at the same point - the CONFIRM step - but the error is now different and is unrecoverable without rolling back to the previous version of firmware on the device.

These are the details:

At the CONFIRM step with our latest firmware version. Attempting to Read the Image data and the error returned is "Encryption is insufficient"
Attempting to Read the Image data again and the error changes to "Connection failed"
After ‘Forgetting the device’ and turning Bluetooth Off and On again on the phone and attempting to Read the Image data again you get prompted to Pair to the device:
The error retuned at this stage is "Authentication is insufficient".
Device Manager thinks it is disconnected but the phone thinks it is connected:
Turning Bluetooth Off and On on the phone and attempting to Read the Image data again and the error is back to Encryption is insufficient and then Connection failed.

Basically at this point it is not possible to connect to the SMP characteristic to CONFIRM the image and on the next device reboot it rolls back to the previous firmware version.

For all previous firmware, and for this firmware also, the SMP Characteristic has been set to have Encryption without Authentication.

We're a bit stuck at this stage, as we basically can't release product because we can't upgrade the firmware.

Regards,

Mike

RE: Issues with completing DFU after updated to NCS V2.6.0

Vidar Berg — Wed, 22 May 2024 11:48:48 GMT

Hi Mike,

Yes, if I recall correctly, you had to register the SMP service from the app, while in version 2.6.0 you can use the Kconfig symbols to have the service registered automatically at startup before main(). This may affect the order in which the services are placed in the attribute table.

You can use the nRF Connect app to check if the services have switched places. However, if you do this, make sure your phone is not bonded and that you turn Bluetooth off and on to avoid the attributes being cached across connections.

E.g.

1. Flash v2.2.0 to your board.
2. Forget the device from settings if it is bonded.
3. Turn Bluetooth off and on to clear any existing cache.
4. Connect to your device using the nRF Connect app and take note of the order in which the services are placed.
5. Repeat steps 1-4 with your new app based on v2.6.1.

Regards,

Vidar

RE: Issues with completing DFU after updated to NCS V2.6.0

Mike Austin (LPI) — Wed, 22 May 2024 11:27:49 GMT

Hi Vidar,

Thanks for the info.

I've not actually changed the service definitions from one version to another, and if I understand how the SMP service is set up, its all bundled in with these two CONFIG settings, at least in NCS V2.6.0:

CONFIG_BOOTLOADER_MCUBOOT=y
CONFIG_NCS_SAMPLE_MCUMGR_BT_OTA_DFU=y

In V2.2.0, I needed to set up most of the CONFIG settings separately.

At this point, I've decided to attempt to make the minor updates I need to keep moving forward with our product using NCS v2.2.0, but I will need to resolve this issue at some point when I do need to update my NCS

Regards,

Mike

RE: Issues with completing DFU after updated to NCS V2.6.0

Vidar Berg — Tue, 21 May 2024 07:23:30 GMT

Hi Mike,

Thanks for the additional information, and sorry for the delayed response. I was out of the office on Friday and Monday as those were public holidays here in Norway.

[quote user="Mike Austin (LPI)"]Go to iPhone Bluetooth Settings and ‘Forget the Device’[/quote]

Instead of removing the bond, could you try enabling logging or debugging the new app to confirm if the link becomes secured? Alternatively, you can confirm this by capturing a sniffer trace.

It could be that the bonding information is not corrupted, but that the SMP service and characteristic have been assigned different attribute handles in the new version. And that for some reason, the app may be trying to access the characteristic through the old handle. The attribute handle assigned to the SMP service depends on the order in which your BLE services are registered in the app.

[quote user="Mike Austin (LPI)"]Alternatively, with the changes to Zephyr and the SDK, is the SMP Characteristic no longer being set to require Authentication??[/quote]

The SMP serivce did not require authentication by default in v2.2.0 either.

Regards,

Vidar

RE: Issues with completing DFU after updated to NCS V2.6.0

Mike Austin (LPI) — Fri, 17 May 2024 00:16:18 GMT

Hi Vidar,

Yep - using the same static partition layout for both versions. Flash utilisation is around 91%

Memory region         Used Size  Region Size  %age Used
           FLASH:       26002 B        48 KB     52.90%
             RAM:       23680 B        64 KB     36.13%
        IDT_LIST:          0 GB        32 KB      0.00%
[34/42] Linking C executable zephyr\zephyr.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:      206412 B     224768 B     91.83%
             RAM:       62408 B        64 KB     95.23%
        IDT_LIST:          0 GB        32 KB      0.00%
[37/42] Generating ../../zephyr/app_update.bin
image.py: sign the payload
[38/42] Generating ../../zephyr/app_signed.hex
image.py: sign the payload
[40/42] Generating ../../zephyr/app_test_update.hex
image.py: sign the payload
[42/42] Generating zephyr/merged.hex

OK, here are the details of what I did to do the upgrade:

Using Confirm Only from the Basic Menu, Upgrade from 1.7 (which was built on V2.2.0) to 1.8 (which was built on V2.6.0) which completes;
Switch to Advanced and UNSUCCESSULLY Read the Image data from the device. The error returned is ‘The handle is invalid’
Go to iPhone Bluetooth Settings and ‘Forget the Device’
Go back to Device Manager Advanced and Successfully read the Image data from the device

I took a bunch of screen shots of the upgrade process as per above.

The selected firmware prior to to starting the Confirm Only upgrade (image1.png)

The Upload Complete state after the upgrade and Reset of the device (image 2.png)

Attempting to Read the Image data immediately after the Upgrade (image 3.png)

Successfully reading the Image data after ‘Forgetting the Device’ in phone Bluetooth Settings (image4.png)

So in summary, the upgrade is completing, but the Pairing/Bonding information appears to be corrupted/lost.

Alternatively, with the changes to Zephyr and the SDK, is the SMP Characteristic no longer being set to require Authentication??

Regards,

Mike

RE: Issues with completing DFU after updated to NCS V2.6.0

Vidar Berg — Thu, 16 May 2024 12:52:12 GMT

Hi Mike,

The confirm action should not require authentication (i.e., bonding) unless the CONFIG_MCUMGR_TRANSPORT_BT_AUTHEN symbol is selected, which does not seem to be the case based on the configuration you posted. Could you please try performing the update from the device manager using the "Confirm only" method from the "basic" menu to see if that enables you to complete the update? This will help narrow down the problem.

Additionally, are you using the same static partition layout for both versions, and what is the flash utilization of the app you are uploading? You can see this from the build log.

Best regards,

Vidar