How to debug MCUboot swap failure after OTA DFU? boot_request_upgrade() has no effect

[QBSho]

Hi everyone,

I'm facing an issue where MCUboot doesn't perform a swap after reboot, even though the DFU download to slot 1 completes successfully and boot_request_upgrade() is called.

My devices are in production and had no issues in the paste with BLE DFU update. As we update to 2.9.0 the problems starts. Some devices are not working after BLE DFU and swap back to the old version. We find it out while implementing OTA over LTE-M that the partition not swapping. There seams an issue with my swapping. The goal is to use the external flash for updating, but if the main flash works it will be good enough again.

Context:

I'm doing firmware updates via OTA DFU, and writing the upgrade request like this:

int err = boot_request_upgrade(BOOT_UPGRADE_TEST); // or int err = boot_set_pending(0);

MCUboot debug logs confirm that both the magic and swap info are written:

<dbg> mcuboot_util: boot_write_magic: writing magic; fa_id=6 off=0x77ff0 (0xfbff0)

<dbg> mcuboot_util: boot_write_swap_info: writing swap_info; fa_id=6 off=0x77fd8 (0xfbfd8), swap_type=0x2 image_num=0x0

However, after reboot, the old image from slot 0 is still running — no swap occurs.

I have CONFIG_MCUBOOT_UTIL_LOG_LEVEL_DBG=y enabled, but I’m unsure how to debug MCUboot itself (e.g., with west debug or bt) since getting not output west attach.

Note on Flash Patch:

I also experimented with:

# CONFIG_DISABLE_FLASH_PATCH=y

When I enable this config, the DFU (BLE) no longer works — so I’ve kept it disabled.

prj.conf (partial):

CONFIG_PARTITION_MANAGER_ENABLED=y

CONFIG_PM_OVERRIDE_EXTERNAL_DRIVER_CHECK=n

CONFIG_PM_EXTERNAL_FLASH_MCUBOOT_SECONDARY=n

CONFIG_PM_PARTITION_REGION_LITTLEFS_EXTERNAL=y

pm_static.yml (simplified):

mcuboot: 0x00000 - 0x0c000

mcuboot_primary: 0x0c000 - 0x84000

mcuboot_secondary: 0x84000 - 0xfc000

settings_storage: 0xfc000 - 0x100000

My questions:

1. What could prevent MCUboot from initiating the swap even when magic/swap_info are written? (Could a mismatch in partition maps between MCUboot and the application prevent the swap?)

2. Is there a correct way to attach to MCUboot for runtime debugging? (e.g., debugging from reset?)

3. Any known issues with OTA DFU + MCUboot + CONFIG_DISABLE_FLASH_PATCH?

4. Are there additional configs I should double-check?

5. How can I verify that MCUboot has the correct view of the flash layout?

Thanks in advance — happy to share more if needed!

[AHaug]

Hi,

What could prevent MCUboot from initiating the swap even when magic/swap_info are written? (Could a mismatch in partition maps between MCUboot and the application prevent the swap?)

Here's a couple of thoughts 

1. Mismatch in size between primary and secondary (both of your partitions are similar if my math checks out)
2. That you don't enough room for swapping (NCSDK-20567 in https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/releases_and_maturity/known_issues.html note that there are two instances of 20567 in this list)
3. That the image is rejected due to either
   1. That the update image is not signed with the same key that MCUboot has (injection protection)
   2. That the update image is rejected due to versioning not being newer than the current version (downgrade protection)

Item 3 is typically occurring after the image has been swapped, so based on your description I don't think that this is the case.

Is there a correct way to attach to MCUboot for runtime debugging? (e.g., debugging from reset?)

 Trying to debug with MCUBOOT enabled fails to boot. and  B0 secure bootloader logging with RTT not working. might be helpful as well as https://academy.nordicsemi.com/courses/nrf-connect-sdk-intermediate/lessons/lesson-2-debugging/ 

Any known issues with OTA DFU + MCUboot + CONFIG_DISABLE_FLASH_PATCH?

Not that I'm aware of, i.e nothing stated in the known issues page and I can't think of any conflicts based on the description https://docs.nordicsemi.com/bundle/ncs-latest/page/kconfig/index.html#CONFIG_DISABLE_FLASH_PATCH 

Are there additional configs I should double-check?

https://academy.nordicsemi.com/courses/nrf-connect-sdk-intermediate/lessons/lesson-8-bootloaders-and-dfu-fota/ there could be some pointers here

How can I verify that MCUboot has the correct view of the flash layout?

MCUboot uses the pm_static.yml that you build it with, so verify it by comparing the partition manager report, i.e west build -t partition_manager_report, up against the generated build partition map that you can find within build/mcuboot/ and/or build/<your_app>

Kind regards,
Andreas

[QBSho]

Thanks Andreas, 

tried all things - but don't get it working!

[AHaug]

QBSho said:

I assume the problem starts with migration to sysbuild. 

Oh, good catch. Do you have a sysbuild.conf? And a sysbuild/mcuboot/ folder? 

Kind regards,
Andreas

[QBSho]

Hello Andreas, 

sysbuild.conf

SB_CONFIG_BOOTLOADER_MCUBOOT=y

sysbuild/mcuboot.overlay

/ {
aliases {
mcuboot-button0 = &button1;
mcuboot-led0 = &red_led;
};
};

sysbuild/mcuboot.conf

CONFIG_MCUBOOT_LOG_LEVEL_WRN=y

I have the feeling that the flash_img_buffered_write in the https://docs.zephyrproject.org/apidoc/latest/group__flash__img__api.html didn't write to the slot1 correctly. Because when i open the nrf connect app and checke the flash it only shows slot 0.

But on the other side when i DFU it works and after confirm only show slot 0 

BR

[AHaug]

Hi,

A couple of items w.r.t your configuraiton

CONFIG_PM_PARTITION_REGION_LITTLEFS_EXTERNAL=y

For this config to be active you need to have CONFIG_PM_OVERRIDE_EXTERNAL_DRIVER_CHECK=y for both images, i.e SB_CONFIG_PM_OVERRIDE_EXTERNAL_DRIVER_CHECK=y in sysbuild.conf 

I see that you're not intending to use external flash for the update image (SB_CONFIG_PM_EXTERNAL_FLASH_MCUBOOT_SECONDARY=n), is this correct?

Secondary:

Could you verify you have a b0 or not? If you are, then this might be a fix for you https://github.com/nrfconnect/sdk-nrf/pull/22454 

Could you examine the mcuboot image headers are included in the hex either with https://docs.nordicsemi.com/bundle/ncs-latest/page/mcuboot/imgtool.html or by following the suggestions here: https://devzone.nordicsemi.com/f/nordic-q-a/111974/mcuboot-loses-pending-update-image-in-slot-1-after-reset?ReplyFilter=Answers&ReplySortBy=Answers&ReplySortOrder=Descending 

Kind regards,
Andreas

[Achim Kraus]

set(PM_STATIC_YML_FILE ${CMAKE_CURRENT_LIST_DIR}/configuration/pm_static.yml CACHE INTERNAL "")

Do you use something as this to load the static partition? Where do you load the "pm_static.yml"?

In the CMakeLists.txt? Then maybe moving that into sysbuild.cmake helps.

[QBSho]

1) CONFIG_PM_PARTITION_REGION_LITTLEFS_EXTERNAL
-> I use external flash for telemetry data not for any mcu relevant thing (no mcuboot on that)

2) CONFIG_PM_OVERRIDE_EXTERNAL_DRIVER_CHECK=y didn't change anything and got a following on SB_ version.
warning: ignoring malformed line 'SB_CONFIG_PM_OVERRIDE_EXTERNAL_DRIVER_CHECK=y'

3) 
Where i find out if b0 or not? in the pm_static.yml or partion.yml is no b0.

4) Double check signed images for both are the same and valid

I got an error `python3 imgtool.py dumpinfo zepyhr.signed.hex , so i guess it's not included

_tlv_prot_head = struct.unpack(
'HH',
b[tlv_off:(tlv_off + image.TLV_INFO_SIZE)])

the bin output 

#### Image header (offset: 0x0) ############################
magic:              0x96f3b83d
load_addr:          0x0
hdr_size:           0x200
protected_tlv_size: 0x0
img_size:           0x74df8
flags:              0x0
version:            0.2.3+0
############################################################
#### Payload (offset: 0x200) ###############################
|                                                          |
|              FW image (size: 0x74df8 Bytes)              |
|                                                          |
############################################################
#### TLV area (offset: 0x74ff8) ############################
magic:     0x6907
area size: 0x97
        ---------------------------------------------
        type: SHA256 (0x10)
        len:  0x20
        data: 0xd8 0x32 0x79 0xeb 0x59 0xee 0x14 0x70 
              0x7b 0x0f 0x80 0xbc 0xde 0x93 0x96 0x3d 
              0x96 0x19 0x90 0x7b 0x49 0x24 0xcf 0x97 
              0x4e 0x22 0x70 0x71 0x85 0x06 0x80 0x5b 
        ---------------------------------------------
        type: KEYHASH (0x1)
        len:  0x20
        data: 0xe3 0x04 0x66 0xf6 0xb8 0x47 0x0c 0x1f 
              0x29 0x07 0x0b 0x17 0xf1 0xe2 0xd3 0xe9 
              0x4d 0x44 0x5e 0x3f 0x60 0x80 0x87 0xfd 
              0xc7 0x11 0xe4 0x38 0x2b 0xb5 0x38 0xb6 
        ---------------------------------------------
        type: ECDSASIG (0x22)
        len:  0x47
        data: 0x30 0x45 0x02 0x20 0x63 0x19 0x8b 0xd1 
              0x92 0x3a 0xdf 0x54 0x90 0xe5 0x80 0x77 
              0x50 0xb2 0xd1 0xab 0x52 0x8f 0x84 0xf7 
              0x4e 0xfb 0x45 0x83 0xb9 0x70 0x8d 0xeb 
              0x3d 0x26 0x9e 0x77 0x02 0x21 0x00 0xcf 
              0xa8 0xc8 0x28 0x46 0x1b 0xb0 0x39 0xa6 
              0x0f 0xe1 0x1c 0xc5 0x94 0x40 0x40 0xde 
              0xd2 0x69 0x72 0x11 0x24 0x07 0x28 0x2a 
              0xe9 0xdf 0x15 0xd3 0x1c 0x52 0xf0 
############################################################
#### End of Image  #########################################
dumpinfo has run successfully

5)
set(PM_STATIC_YML_FILE ${CMAKE_CURRENT_LIST_DIR}/configuration/pm_static.yml CACHE INTERNAL "")

I don't set anything here