Issues with completing DFU after updated to NCS V2.6.0

[Mike Austin (LPI)]

We have some custom hardware that utilises an NRF52832.  Up until recently, all our firmware was built on NCS V2.2.0, which I believe used Zephyr V3.2.99.  We have implemented DFU for firmware upgrades, and haven't had any issues with this.  Up until now.

The latest version of our firmware has been built around NCS V2.6.0, which I think uses Zephyr V3.5.99.  There were a bunch of CONFIG setting changes that I need to make in the transition from NCS V2.2.0 to NCS V2.6.0 associated with setting up the DFU functionality.

We have a custom App that we can use for customers to complete a firmware upgrade.  That initiates the transfer of the new image file into the spare slot, resets the device at the completion, and then awaits a response from the SMP service (the App has subscribed to the SMP Service notifications) to tell it that the new image has been accepted and the device has booted up using the new image.

For whatever reason, that notification from the device to the App via the SMP service is no longer occurring, so our App sits there endlessly waiting.

When we try and test the same functionality using Nordic's Device Manager App, we see the following:

• New image uploads into Slot 1 OK
• READ - this confirms the old image in Slot 0 and the new, pending image, in Slot 1
• TEST - this confirms the new image in Slot 1 as the one to boot up from next time the device starts up
• RESET - this works as expected
• CONFIRM - this fails, with either an error message that says "Insufficient handle" or something about insufficent authentication

Basically, it looks like pairing/bonding information is getting wiped in the DFU process.  But this authentication either wasn't needed when we were doing DFU with V2.2.0, or the pairing/bonding info wasn't getting erased with the new image upload.

I suspect there is some CONFIG setting that is now being set that previously wasn't.  I've attached my proj.conf file so you can see how I currently have things configured

# Operational CONFIG settings

# General
CONFIG_REBOOT=y
CONFIG_GPIO=y
CONFIG_BOARD_ENABLE_DCDC=n
CONFIG_ADC=y

# Logging
CONFIG_LOG=y
CONFIG_LOG_MODE_MINIMAL=n
CONFIG_LOG_DEFAULT_LEVEL=0

# Boot
CONFIG_BOOT_BANNER=n

#Device Name - will be visible to clients scanning
CONFIG_BT_DEVICE_NAME="My Device"

# Bluetooth
CONFIG_BT=y
CONFIG_BT_PERIPHERAL=y
CONFIG_BT_SMP=y
CONFIG_BT_GATT_CLIENT=y
CONFIG_BT_GATT_DM=y
CONFIG_BT_MAX_PAIRED=10
CONFIG_BT_ID_MAX=10
CONFIG_BT_ID_UNPAIR_MATCHING_BONDS=y
CONFIG_BT_KEYS_OVERWRITE_OLDEST=y
CONFIG_BT_SMP_SC_PAIR_ONLY=y
#CONFIG_BT_CTS_CLIENT=y

#PHY update needed for updating PHY request
CONFIG_BT_USER_PHY_UPDATE=y
#For data length update
CONFIG_BT_USER_DATA_LEN_UPDATE=y
CONFIG_BT_BUF_ACL_TX_COUNT=10
CONFIG_BT_ATT_PREPARE_COUNT=2

# Enable CTS client
CONFIG_BT_CTS_CLIENT=y

# Below is setup to let DIS information be read from settings
CONFIG_BT_SETTINGS=y
CONFIG_SETTINGS_RUNTIME=y
CONFIG_SETTINGS=y

#Enable MCUBOOT bootloader build in the application
CONFIG_BOOTLOADER_MCUBOOT=y
#Include MCUMGR and the dependencies in the build
CONFIG_NCS_SAMPLE_MCUMGR_BT_OTA_DFU=y
CONFIG_NCS_SAMPLE_MCUMGR_BT_OTA_DFU_SPEEDUP=y
CONFIG_NCS_SAMPLE_MCUMGR_BT_OTA_DFU_VALIDATION=y

# Set stack and heap sizes
CONFIG_MAIN_STACK_SIZE=2048
CONFIG_HEAP_MEM_POOL_SIZE=2048

# Allow flash
CONFIG_FLASH=y
CONFIG_FLASH_PAGE_LAYOUT=y
CONFIG_FLASH_MAP=y
CONFIG_NVS=y
CONFIG_MPU_ALLOW_FLASH_WRITE=y

# Enable Power Management
CONFIG_POWEROFF=y

# Required to disable default behavior of deep sleep on timeout
CONFIG_PM_DEVICE=y

# Enable I2C
CONFIG_I2C=y
CONFIG_COUNTER=y
CONFIG_PCF85063A=y

#cJSON
CONFIG_CJSON_LIB=y
CONFIG_NEWLIB_LIBC=y
CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y

# Sensors
CONFIG_SENSOR=y

# CONFIG below here set up to minimise flash usage

# Disable features not needed
CONFIG_TIMESLICING=n
CONFIG_ASSERT=n

# Disable Bluetooth features not needed
CONFIG_BT_DEBUG_NONE=y
CONFIG_BT_ASSERT=n
CONFIG_BT_GATT_CACHING=n
CONFIG_BT_SETTINGS_CCC_LAZY_LOADING=y
CONFIG_BT_HCI_VS_EXT=n

# Disable Bluetooth controller features not needed
CONFIG_BT_CTLR_PRIVACY=n
CONFIG_BT_CTLR_PHY_2M=n

# Reduce Bluetooth buffers
CONFIG_BT_BUF_EVT_DISCARDABLE_COUNT=1
CONFIG_BT_BUF_EVT_RX_COUNT=2

# Drivers and peripherals
CONFIG_WATCHDOG=n
CONFIG_SPI=n

# Interrupts
CONFIG_DYNAMIC_INTERRUPTS=n

# Memory protection
CONFIG_THREAD_STACK_INFO=n
CONFIG_THREAD_CUSTOM_DATA=n
CONFIG_FPU=n

# Console
CONFIG_EARLY_CONSOLE=n

# Build
CONFIG_SIZE_OPTIMIZATIONS=y

Can anyone assist me in resolving this issue?  We need to roll out the updated version of firmware, but can't until we can ensure a smooth upgrade process through our App.

Regards,

Mike

[Vidar Berg]

Hi Mike,

The confirm action should not require authentication (i.e., bonding) unless the CONFIG_MCUMGR_TRANSPORT_BT_AUTHEN symbol is selected, which does not seem to be the case based on the configuration you posted. Could you please try performing the update from the device manager using the "Confirm only" method from the "basic" menu to see if that enables you to complete the update? This will help narrow down the problem.

Additionally, are you using the same static partition layout for both versions, and what is the flash utilization of the app you are uploading? You can see this from the build log.

Best regards,

Vidar

[Mike Austin (LPI)]

Hi Vidar,

Yep - using the same static partition layout for both versions.  Flash utilisation is around 91%

Memory region         Used Size  Region Size  %age Used
           FLASH:       26002 B        48 KB     52.90%
             RAM:       23680 B        64 KB     36.13%
        IDT_LIST:          0 GB        32 KB      0.00%
[34/42] Linking C executable zephyr\zephyr.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:      206412 B     224768 B     91.83%
             RAM:       62408 B        64 KB     95.23%
        IDT_LIST:          0 GB        32 KB      0.00%
[37/42] Generating ../../zephyr/app_update.bin
image.py: sign the payload
[38/42] Generating ../../zephyr/app_signed.hex
image.py: sign the payload
[40/42] Generating ../../zephyr/app_test_update.hex
image.py: sign the payload
[42/42] Generating zephyr/merged.hex

OK, here are the details of what I did to do the upgrade:

• Using Confirm Only from the Basic Menu, Upgrade from 1.7 (which was built on V2.2.0) to 1.8 (which was built on V2.6.0) which completes;
• Switch to Advanced and UNSUCCESSULLY Read the Image data from the device.  The error returned is ‘The handle is invalid’
• Go to iPhone Bluetooth Settings and ‘Forget the Device’
• Go back to Device Manager Advanced and Successfully read the Image data from the device

I took a bunch of screen shots of the upgrade process as per above.

The selected firmware prior to to starting the Confirm Only upgrade (image1.png)

The Upload Complete state after the upgrade and Reset of the device (image 2.png)

Attempting to Read the Image data immediately after the Upgrade (image 3.png)

Successfully reading the Image data after ‘Forgetting the Device’ in phone Bluetooth Settings (image4.png)

So in summary, the upgrade is completing, but the Pairing/Bonding information appears to be corrupted/lost. 

Alternatively, with the changes to Zephyr and the SDK, is the SMP Characteristic no longer being set to require Authentication??

Regards,

Mike

[Vidar Berg]

Hi Mike,

Thanks for the additional information, and sorry for the delayed response. I was out of the office on Friday and Monday as those were public holidays here in Norway. 

Mike Austin (LPI) said:

Go to iPhone Bluetooth Settings and ‘Forget the Device’

Instead of removing the bond, could you try enabling logging or debugging the new app to confirm if the link becomes secured? Alternatively, you can confirm this by capturing a sniffer trace.

It could be that the bonding information is not corrupted, but that the SMP service and characteristic have been assigned different attribute handles in the new version. And that for some reason, the app may be trying to access the characteristic through the old handle. The attribute handle assigned to the SMP service depends on the order in which your BLE services are registered in the app.

Mike Austin (LPI) said:

Alternatively, with the changes to Zephyr and the SDK, is the SMP Characteristic no longer being set to require Authentication??

The SMP serivce did not require authentication by default in v2.2.0 either.

Regards,

Vidar

[Mike Austin (LPI)]

Hi Vidar,

Thanks for the info.

I've not actually changed the service definitions from one version to another, and if I understand how the SMP service is set up, its all bundled in with these two CONFIG settings, at least in NCS V2.6.0:

CONFIG_BOOTLOADER_MCUBOOT=y
CONFIG_NCS_SAMPLE_MCUMGR_BT_OTA_DFU=y

In V2.2.0, I needed to set up most of the CONFIG settings separately.

At this point, I've decided to attempt to make the minor updates I need to keep moving forward with our product using NCS v2.2.0, but I will need to resolve this issue at some point when I do need to update my NCS

Regards,

Mike

[Vidar Berg]

Hi Mike,

Yes, if I recall correctly, you had to register the SMP service from the app, while in version 2.6.0 you can use the Kconfig symbols to have the service registered automatically at startup before main(). This may affect the order in which the services are placed in the attribute table.

You can use the nRF Connect app to check if the services have switched places. However, if you do this, make sure your phone is not bonded and that you turn Bluetooth off and on to avoid the attributes being cached across connections.

E.g.

1. Flash v2.2.0 to your board.
2. Forget the device from settings if it is bonded.
3. Turn Bluetooth off and on to clear any existing cache.
4. Connect to your device using the nRF Connect app and take note of the order in which the services are placed.
5. Repeat steps 1-4 with your new app based on v2.6.1.

Regards,

Vidar

[Mike Austin (LPI)]

Hi Vidar,

OK, checked the Attributes Table order between the two versions of firmware built around the two versions of NCS, and they are exactly the same.

So, something seems to have changed in how the SMP service is set up.

To try and get around this in the short term (we have product that customers are unable to update the firmware for, so I'm trying to get something that will work), I did the following:

1. Went back to NCS v2.2.0/Zephyr v3.2.99
2. Took the version of firmware that used to work, which was v1.7.  This successfully upgrades from v1.6 via our custom app.
3. Made the very minor changes that I need to make to sort out the bug we have uncovered
4. Set that as v1.9 (v1.8 is the one built around NCS v2.6.0 that we have been having issues with) and did a pristine build, using NCS v2.2.0
5. Attempted to do the upgrade via the Device Manager

Testing using Device Manager, we are now seeing something similar to before, even though we aren't changing the version of NCS/Zephyr now!  The steps are the same, and the failure occurs at the same point - the CONFIRM step - but the error is now different and is unrecoverable without rolling back to the previous version of firmware on the device.

These are the details:

1. At the CONFIRM step with our latest firmware version.  Attempting to Read the Image data and the error returned is "Encryption is insufficient"
   
   
   
2. Attempting to Read the Image data again and the error changes to "Connection failed"
   
3. After ‘Forgetting the device’ and turning Bluetooth Off and On again on the phone and attempting to Read the Image data again you get prompted to Pair to the device:
   
4. The error retuned at this stage is "Authentication is insufficient".
   
5. Device Manager thinks it is disconnected but the phone thinks it is connected:
6. Turning Bluetooth Off and On on the phone and attempting to Read the Image data again and the error is back to Encryption is insufficient and then Connection failed.

 

Basically at this point it is not possible to connect to the SMP characteristic to CONFIRM the image and on the next device reboot it rolls back to the previous firmware version.

For all previous firmware, and for this firmware also, the SMP Characteristic has been set to have Encryption without Authentication.

We're a bit stuck at this stage, as we basically can't release product because we can't upgrade the firmware.

Regards,

Mike