DFU using USB file system (not serial emulation)?

[BrianW]

My project has a nrf5340+external flash, and an external USB-C connector. The device exposes a USB file system (FAT) which is hosted on the external flash partition, and appears as an external disk when connected to a host. This all works well and is used to update config, image files etc for the device to use.

I would like to be able to do application / network firmware updates by simply copying the new image (hex or bin) to this file system and rebooting. Has anyone done this? I'm guessing it involves using MCUBoot bootloader and copying the image into the secondary image slot somehow? I also want to put the secondard image slot on my external flash as won't have space on the internal flash - even better would be if MCUBoot could take the image directly from the file in the FS....

by the way, to be clear, I do NOT want to use the 'DFU over USB serial emulation' method, as this requires the user to install the specific tool to the host machine, which may not be available for their host or may not be allowed by IT policy (or by their level of expertise...). 

thanks for any pointers!

[Hung Bui]

Hi Brian, 
You should set 

CONFIG_MCUBOOT_SERIAL=n in the mcuboot.conf overlay to apply the config to the MCUBoot child image, not the application. Usually you put that file into child_image folder.  
Anyway , by default MCUBOOT_SERIAL is not enable so you don't really have to worry about it. (double check with .config file in the MCUBoot build folder)

[BrianW]

Ok... I removed all the wifi stuff from my app, added a mcuboot.cong in the child_image folder with:

CONFIG_MCUBOOT_SERIAL=n

CONFIG_BOOT_USE_MIN_PARTITION_SIZE=y

This didn't fix it, although it has changed things.

Latest build error is:

[373/415] Performing build step for 'mcuboot_subimage'

...

`text' will not fit in region `FLASH'
c:/ncs/toolchains/cf2149caf2/opt/zephyr-sdk/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd.exe: region `FLASH' overflowed by 37320 bytes
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

...
[396/415] Linking C executable zephyr\zephyr_pre0.elf
c:/ncs/toolchains/cf2149caf2/opt/zephyr-sdk/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd.exe: warning: orphan section `.fonts' from `app/libapp.a(u8g2_fonts.c.obj)' being placed in section `.fonts'
[397/415] Generating ../../zephyr/net_core_app_test_update.hex
image.py: sign the payload
[402/415] Linking C executable zephyr\zephyr.elf
FAILED: modules/mcuboot/mcuboot_subimage-prefix/src/mcuboot_subimage-stamp/mcuboot_subimage-build mcuboot/zephyr/zephyr.hex mcuboot/zephyr/zephyr.elf C:/work/dev/if-device-nrf53/cc1-med/build/modules/mcuboot/mcuboot_subimage-prefix/src/mcuboot_subimage-stamp/mcuboot_subimage-build C:/work/dev/if-device-nrf53/cc1-med/build/mcuboot/zephyr/zephyr.hex C:/work/dev/if-device-nrf53/cc1-med/build/mcuboot/zephyr/zephyr.elf
...
Memory region Used Size Region Size %age Used
FLASH: 446762 B 1031680 B 43.30%
RAM: 152572 B 440 KB 33.86%
IDT_LIST: 0 GB 32 KB 0.00%
ninja: build stopped: subcommand failed.

I note that the application built correctly, and is around 446Kb (with the wifi enabled I'm thinking this was showing 901kB...). But this didnt stop the mcuboot subimage failing, this time with its 'text' segment overflowing the FLASH...

Maybe I should try to reconfigure to stop using the partition manager if its creating a partition for mcuboot which is too small?

or just build the mcuboot child image to check if that is the issue?

[Hung Bui]

Hi Brian, 

Sorry for the late response. 
I took a look at your result again. It seems that what you have: 
I: Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x1
I: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3

Matched with this scenario: 
{
.magic_primary_slot = BOOT_MAGIC_ANY,
.magic_secondary_slot = BOOT_MAGIC_GOOD,
.image_ok_primary_slot = BOOT_FLAG_ANY,
.image_ok_secondary_slot = BOOT_FLAG_UNSET,
.copy_done_primary_slot = BOOT_FLAG_ANY,
.swap_type = BOOT_SWAP_TYPE_TEST,
},

You have magic_secondary_slot = good,  image_ok_secondary =0x3 (UNSET)  the rest is just any so it match and it explains why the swap type is test instead of non and it will be swapped. 

Here is what I have in my test, first when I send the test image: 

It matched with the scenario above => swap test.

Then if I don't do anything, it will revert: 

{

        .magic_primary_slot =       BOOT_MAGIC_GOOD,

        .magic_secondary_slot =     BOOT_MAGIC_UNSET,

        .image_ok_primary_slot =    BOOT_FLAG_UNSET,

        .image_ok_secondary_slot =  BOOT_FLAG_ANY,

        .copy_done_primary_slot =   BOOT_FLAG_SET,

        .swap_type =                BOOT_SWAP_TYPE_REVERT,

    },

If I confirm the image instead, next boot is like this: 

So it's very important that the secondary image's magic should be unset and the primary image_ok is set. 
What I noticed in my case is that at the beginning before the first test swap, the primary image's magic is unset. This is very important. The primary image magic will be come the secondary image magic after the swap (it follows the image). So if at the beginning you already set the primary image magic to set, it may result in what you observed. 
The only difference in my no swap booting is the secondary magic is unset (and swaptype =0x01 but  that's not important in my opinion). 
Please take a look and check why you have both image magic = good. 

[BrianW]

I don't think I explicitly set the magic to good anywhere... the code only does the call to say 'image ok' once it runs...

And if we have done a successful update (secondary->primary), the primary will then have a good magic (because it was a good secondary), so for the next update both primary and secondary have magic=good...

I thought the magic was just to let mcuboot know that the image is a legitimate mcuboot image, and its set by the build process?

[BrianW]

So... I come back to this dfu issue after several weeks (months) of having fun migrating to NCS2.9, dealing with sysbuild etc....

Now, I have 3 image slots, for the app, CPU-NET (its a nrf5340) and the wifi firmware (because the wifi code is so big I needed to push the wifi fw (78kb!) out to my external flash).

First moan :  the dfu_target_xx subsystem (supposed to hide the bootloader from me) lacks both a 'check if image is confirmed' and a 'confirm the primary slot of an image is good' method. So I have to go directly to the mcuboot methods to do this anyway (but we already noted the failing of the dfu_target API in the thread above...) 

Second moan : the mcuboot api (exposed in include/zephyr/dfu/mcuboot.h) contains a multi-image compatible call to confirm an image (boot_write_img_confirmed_multi(image_number)), but not a function to check if the image is already confirmed (to avoid writing to the mcuboot header at every boot...): there is boot_is_img_confirmed() but not boot_is_img_confirmed_multi(imgage_number). WHY OH WHY? and there isn't even the function in bootutil.c (which mcuboot.c delegates all the multi-image stuff to...)

So I have to add my own function  to bootloader/mcuboot/boot/bootutil/src/bootutil_public.c (as this is the code that has the mapping table for the image number->flash area id....). Not great for future NCS updates....

Now, with sysbuild when I flash the merged.hex, the initial state of the slots is

*** Booting MCUboot v2.1.0-dev-12e5ee106034 ***
*** Using nRF Connect SDK v2.9.0-7787b2649840 ***
*** Using Zephyr OS v3.7.99-1f8f3dc29142 ***
Starting bootloader

Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 0, Swap type: none

Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 1, Swap type: none

Primary image: magic=good, swap_type=0x0, copy_done=0x1, image_ok=0x1
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 2, Swap type: none

Bootloader chainload address offset: 0x11000

Only the primary slots for images 0 and 2 are relevant, as they have been written from the merged.hex.

Immediately we can see that the build sets the magic of the application image (0) primary slot to be UNSET, while the nrf70 wifi fw (2) primary slot has magic=good!

But lets see if updating the NCS2.9 has fixed the mcuboot 'confirmation' issue. I remove my 'break the secondary image code', and make it confirm att 3 slots as 'ok' after 30s of execution....

When I DFU the wifi fw in image 2, it all works ok :

*** Booting MCUboot v2.1.0-dev-12e5ee106034 ***
*** Using nRF Connect SDK v2.9.0-7787b2649840 ***
*** Using Zephyr OS v3.7.99-1f8f3dc29142 ***
Starting bootloader

Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 0, Swap type: none

Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 1, Swap type: none

Primary image: magic=good, swap_type=0x0, copy_done=0x1, image_ok=0x1
Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 2, Swap type: test
Starting swap using move algorithm.

applicaiton then runs, my code then confirms all images as ok using the mcuboot function aftre 30s.

on next reboot:

*** Using Zephyr OS v3.7.99-1f8f3dc29142 ***
Starting bootloader

Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 0, Swap type: none

Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 1, Swap type: none

Primary image: magic=good, swap_type=0x0, copy_done=0x1, image_ok=0x1
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 2, Swap type: none

The image 2 secondary slot has had its magic 'unset' and the swap_type set to NONE (1). So, no reverting. 

BUT: when I do exactly the same operation for the application, the 'confirmation' does not work:

*** Booting MCUboot v2.1.0-dev-12e5ee106034 ***
*** Using nRF Connect SDK v2.9.0-7787b2649840 ***
*** Using Zephyr OS v3.7.99-1f8f3dc29142 ***
Starting bootloader

Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x1
Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 0, Swap type: test

Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 1, Swap type: none

Primary image: magic=good, swap_type=0x0, copy_done=0x1, image_ok=0x1
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 2, Swap type: none

Starting swap using move algorithm.
Bootloader chainload address offset: 0x11000

-> update done ok
after 30s my code confirms the images, exactly as it did for the wifi fw.
[00:00:30.417,694] <inf> app: Boot : confirmed image 0 ok to mcuboot
[00:00:30.417,816] <wrn> app: Boot : failed to confirm mcuboot image 1 as ok (-5)
[00:00:30.418,182] <inf> app: Boot : image 2 is already flagged as ok to mcuboot

reboot:

*** Booting MCUboot v2.1.0-dev-12e5ee106034 ***
*** Using nRF Connect SDK v2.9.0-7787b2649840 ***
*** Using Zephyr OS v3.7.99-1f8f3dc29142 ***
Starting bootloader

Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x1
Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 0, Swap type: test

Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 1, Swap type: none

Primary image: magic=good, swap_type=0x0, copy_done=0x1, image_ok=0x1
Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
Boot source: none
Image index: 2, Swap type: none
Starting swap using move algorithm.

And we're back to flip-flop between the 2 slots at each reboot.

In the wifi case, I don't  see what code has set the secondary slot's magic to unset, which is the differnce between the wifi and the app cases... I'm wondering if its actually the wifi setup firmware loader code?

Any ideas?

[BrianW]

Hung Bui said:

If I confirm the image instead, next boot is like this: 

So it's very important that the secondary image's magic should be unset and the primary image_ok is set. 
What I noticed in my case is that at the beginning before the first test swap, the primary image's magic is unset. This is very important. The primary image magic will be come the secondary image magic after the swap (it follows the image). So if at the beginning you already set the primary image magic to set, it may result in what you observed. 

Having read thru the mcuboot bootutil_public.c boot_swap_type_multi() code, which uses the boot swap tables arrayu to decide what should happen at boot time, I agree that if the secondary magic is GOOD, then it WILL be used to swap (either as test or perm). So, as after the DFU the secondary is the ex-primary, this implies the PRIMARY must have magic=UNSET (means all 0xff for empty flash) before we start DFU.

If I look at you 'post successful DFU' screenshot, I note the primary has magic=GOOD -> so if you do the DFU step again in future, the secondary will end up with good magic, and the DFU will rollback!

So.... who/where/how should the magic of the current primary be cleared? I would think this could be done when the code confirms that the new DFU target is in place in the secondary slot?

This is the call that does that

dfu_target_mcuboot.c : dfu_target_mcuboot_schedule_update(img_num);

-> calls 

dfu_target_mcuboot.c : dfu_target_mcuboot_schedule_one_img(img);

-> calls 

mcuboot.c : boot_request_upgrade_multi(img_num, BOOT_UPGRADE_TEST);

-> calls (as BOOT_UPGRADE_TEST=0)

bootutil_public. : boot_set_pending_multi(img, 0);

-> calls

bootutil_public. : boot_set_next(<flash area of secondary slot for img>, false, false); -> this is not the active image, and we don't confirm it

In here, if your secondary slot magic is already GOOD, it does... nothing.

As we have just updated the secondary slot with the new firmware using dfu_target, the magic should be UNSET.

In this case, it writes the good magic, and leaves image ok as unset (because its not a confirmed ie permanent move). It also writes the 'swap_type' in the trailer to TEST based on this flag.

It does NOT do anything to the primary's magic slot....

 

Do you agree therefore that this will work for a DFU once, but the 2nd time we have the magic=good on the primary and this will cause the flip flop for subsequent cases? Or do I miss something I should do? 

[BrianW]

So.... a CRITICAL thing I have discovered, which is not IMHO well documented, is that it is essential that your slot partitions for mcuboot (primary, secondary) are both aligned at their start address on a flash erase sector boundry AND that their size is ALSO aligned on a erase boundry (0x1000=4Kb for my flash device for example).

The 'mcuboot_pad' partition (for the 1st image ie the app) is the one to align on the boundry, not the following 'app' partition (so that the 'mcuboot_primary' partition ends up on the boundary). 

Its the 'mcuboot_primary' size that must be aligned, and the 'mcuboot_secondary' must be this size, not the app size....

If you don't align the start of the particion : after a mcuboot swap, the resulting primary image is unreadable for booting (presumably because it couldn't erase the start sector properly?)

If you don't align the size of the primary/secondary : the swap procedure fails to clear the mcuboot trailer (because its size is not aligned for erase?), and your 2 slots will flip-flop on reboot....

When you are trying to use every last byte of the internal flash for your app (eg because the wifi stack is taking up 3/4 of the flash....) then the temptation is to avoid any wasted space... but the alignment needs to be there as the mcuboot code doesn't cope otherwise....

Here's my new pm_static.yml : note the 'app' is not an aligned size, but both mcuboot_primary and mcuboot_secondary are...

mcuboot:
    address: 0x0
    end_address: 0x11000
    region: flash_primary
    affiliation: 
        - mcuboot
    size: 0x11000

mcuboot_pad:
    # should be aligned on CONFIG_FPROTECT_BLOCK_SIZE (0x4000 on nrf5340) boundry to allow FPROTECT to be enabled
    # and AT minimum on 0x1000 sector size for erase needs (otherwise DFU (move algo) will NOT WORK)
    address: 0x11000
    end_address: 0x11200
    region: flash_primary
    affiliation: 
        - mcuboot
    size: 0x200

app:
    address: 0x11200
    end_address: 0x100000
    region: flash_primary
    size: 0xeee00

mcuboot_primary_app:
    address: 0x11200
    end_address: 0x100000
    region: flash_primary
    affiliation: 
        - mcuboot
    size: 0xeee00
    span: [app]

mcuboot_primary:
    address: 0x11000
    end_address: 0x100000
    region: flash_primary
    affiliation: 
        - mcuboot
    size: 0xef000
    span: [mcuboot_pad, app]

# define partition for fatfs disk, on the external flash storage (probably the device defined by nordic,pm-ext-flash in the DTS)
fatfs_storage:
    region: external_flash
    affiliation: 
        - disk
    extra_params: {
        disk_name: "NAND",
        disk_cache_size: 4096,
        disk_sector_size: 512,
        disk_read_only: 0
    }
    # 6Mb size, external flash is 8Mb total
    address: 0x0
    size: 0x600000

nvs_storage:
    region: external_flash
    affiliation: 
        - nvs
    # 64kB size, its just for small stuff
    address: 0x600000
    size: 0x10000

mcuboot_secondary_1:
    region: external_flash
    affiliation: 
        - mcuboot
    # 256kb size, its for netcore flash image
    address: 0x610000
    size: 0x40000

mcuboot_secondary:
    region: external_flash
    affiliation: 
        - mcuboot
    address: 0x650000
    # MUST be same size as the 'mcuboot_primary' slot size, and share_size does not appear to work
    size: 0xef000
    #share_size: [mcuboot_primary]

# get next slot to be aligned on 0x4000
PAD1_END:
    region: external_flash
    address: 0x73f000
    size: 0x1000

# allow wifi patches to be updated by mcuboot
nrf70_wifi_fw_mcuboot_pad:
    region: external_flash
    address: 0x740000
    size: 0x200
    #device: MX25R64

nrf70_wifi_fw:
    region: external_flash
    # 128kB size for the wifi fw (-0x200 bytes, the mcuboot pad so that total is aligned on 0x1000 - and 0x4000 - to keep mcuboot happy), 
    address: 0x740200
    size: 0x1fe00

mcuboot_primary_2:
    region: external_flash
    orig_span: &id003
    - nrf70_wifi_fw_mcuboot_pad
    - nrf70_wifi_fw
    span: *id003
    address: 0x740000
    affiliation: 
        - mcuboot
    size: 0x20000
    #device: MX25R64

mcuboot_secondary_2:
    region: external_flash
    address: 0x760000
    affiliation: 
        - mcuboot
    size: 0x20000
    #device: MX25R64

# remaining external flash space starts at 0x78000

# need fake flash ram flash partition to do cpunet DFU...
mcuboot_primary_1:
    region: ram_flash
    address: 0x0
    affiliation: 
        - mcuboot
    size: 0x40000
    device: flash_ctrl
    #  end_address: 0x40000
    #  device: nordic_ram_flash_controller

ram_flash:
  address: 0x40000
  end_address: 0x40000
  region: ram_flash
  size: 0x0

# internal cuisine
#otp:
#    address: 0xff8100
#    end_address: 0xff83fc
#    region: otp
#    size: 0x2fc
#pcd_sram:
#    address: 0x20000000
#    end_address: 0x20002000
#    region: sram_primary
#    size: 0x2000
#rpmsg_nrf53_sram:
#    address: 0x20070000
#    end_address: 0x20080000
#    region: sram_primary
#    size: 0x10000
#sram_primary:
#    address: 0x20002000
#    end_address: 0x20070000
#    region: sram_primary
#    size: 0x6e000

And with this layout that critically aligned the size of the mcuboot_secondary on a flash erase boundary, the mcuboot 'swap-using-move' procedure plus the call to boot_write_img_confirmed_multi() mean that the image no longer flip flops on each boot! 

This should really be in the mcuboot/bootloader doc as its a critical point and not obvious....

by the way, it would be nice if in the dfu_target subsystem dfu_target_mcuboot.c provided the required multi-image confirmation functions

dfu_target_mcuboot_is_img_confirmed_multi(int img_num)

dfu_target_mcuboot_img_confirm_multi(int img_num)