DFU using USB file system (not serial emulation)?

Question

My project has a nrf5340+external flash, and an external USB-C connector. The device exposes a USB file system (FAT) which is hosted on the external flash partition, and appears as an external disk when connected to a host. This all works well and is used to update config, image files etc for the device to use. 
 I would like to be able to do application / network firmware updates by simply copying the new image (hex or bin) to this file system and rebooting. Has anyone done this? I'm guessing it involves using MCUBoot bootloader and copying the image into the secondary image slot somehow? I also want to put the secondard image slot on my external flash as won't have space on the internal flash - even better would be if MCUBoot could take the image directly from the file in the FS.... 
 by the way, to be clear, I do NOT want to use the 'DFU over USB serial emulation' method, as this requires the user to install the specific tool to the host machine, which may not be available for their host or may not be allowed by IT policy (or by their level of expertise...). 
 thanks for any pointers!

BrianW · Answer

BrianW said:
but it doesn't mention having to set a buffer for mcuboot before the call to dfu_target_init()

And nowhere in dfu_target.c does it do this?!

What is the point of having a 'standard' api dfu_target to mask the different 'targets', if app has to explicitly call a target specific function to get it to work? And not described in the doc either?

So... I changed my code to just use the dfu_target_mscuboot_XXX API directly. This all seems to work just fine, except that after dfu_target completes (with no errors), mcuboot still refuses to update from the secondary slot : the relevant logs:

[00:00:00.544,769] <inf> base: dfu: opened app_update.bin, image type identified as mcuboot
[00:00:00.555,236] <inf> base: dfu: starting update of image for app_update.bin
Copying to secondard slot.................................................................................................................................................................................................................................. DONE
[00:00:27.828,277] <inf> dfu_target_mcuboot: MCUBoot image-0 upgrade scheduled. Reset device to apply
[00:00:27.838,562] <inf> base: dfu: no image file for net_core_app_update.bin
[00:00:27.846,130] <wrn> base: dfu : found new images, rebooting in 5s to update....see you on the other side...
Rebooting...

*** Booting nRF Connect SDK 3758bcbfa5cd ***
I: Starting bootloader
W: Cannot upgrade: not a compatible amount of sectors
I: Bootloader chainload address offset: 0xe000

After a search on this forum, I found this:

https://devzone.nordicsemi.com/f/nordic-q-a/94313/mcuboot-runtime-error-cannot-upgrade-not-a-compatible-amount-of-sectors-with-external-flash-on-nrf9160

Turns out that the size of the secondard slot MUST be exactly equal to the primary (not just bigger), and that a) this is not enforced at build time and b) the share_size directive does not work in your pm_static.yml.

[as an aside, I note that post was over 2 years ago, and the information about these 2 points was supposed to being fed back to developers.... but still no fix or even any documentation....]

Making my mcuboot_secondary partition be the same size as the mcuboot_primary_app one fixed the mcuboot log and now I get:

............................................................................................................................................................................................................................ DONE
[00:00:27.582,580] <inf> dfu_target_mcuboot: MCUBoot image-0 upgrade scheduled. Reset device to apply
[00:00:27.592,864] <inf> base: dfu: no image file for net_core_app_update.bin
[00:00:27.600,402] <inf> base: dfu :System write boot ok flag to NVM
[00:00:27.607,360] <wrn> base: dfu : found new images, rebooting in 5s to update....
[00:00:30.319,671] <inf> base: System running ok, write boot ok flag to NVM
Rebooting...see you on the other side...

*** Booting nRF Connect SDK 3758bcbfa5cd ***
I: Starting bootloader
I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Image index: 0, Swap type: test
I: Starting swap using move algorithm.
I: Bootloader chainload address offset: 0xe000

And miracle : a working DFU from USB filesystem!

For those also looking to do this:

the 'dfu from local file' fn (call it with the name of the file on the FS, and reboot if it returns true). You will need to implement the usb_fs_XX functions, but these are just wrappers round the FAT FS fs_XX api.

// DFU related code
static void _dfu_cb(enum dfu_target_evt_id evt_id) {
	log_info("dfu: callback says event %d", evt_id);
}

// read blocks of 4K from fle and pass to DFU updater
// Returns false if couldn't access or dfu the given file 
#define DFU_BUF_SZ	(4096)
static bool _do_dfu(char* filename, int img_num) {
	int32_t fsize = usb_fs_get_file_size(filename);
	if (fsize<0) {
		log_info("dfu: no image file for %s",filename);
		return false;
	}
	file_handle_t fh = usb_fs_open_file(filename, true);	// open in read only
	if (fh==NULL) {
		log_warn("dfu : failed to open file %s", filename);
		return false;
	}
	uint8_t* buf = Util_amalloc(DFU_BUF_SZ);
	if (buf==NULL) {
		usb_fs_close_file(fh);
		log_warn("dfu : failed to alloc buffer to read %s", filename);
		return false;
	}
	uint32_t buf_sz = 0;
	// Read first block to let dfu analyse file type
	buf_sz = usb_fs_read_file(fh, buf, DFU_BUF_SZ);
	if (buf_sz!=DFU_BUF_SZ) {
		// image file must be > 4K so if fail to read then give up
		usb_fs_close_file(fh);
		Util_afree(buf);
		log_warn("dfu : failed to read 4K from file %s, %d", filename, buf_sz);
		return false;
	}
	if (!dfu_target_mcuboot_identify(buf)) {
		usb_fs_close_file(fh);
		Util_afree(buf);
		log_warn("dfu : file %s is not an mcuboot image, cannot DFU", filename);
		return false;
	}
	log_info("dfu: opened %s, image type identified as mcuboot", filename);
	dfu_target_mcuboot_reset();	// always start from beginning as not downloading

	// Give it a scratch buffer for flash writing
	uint8_t* mcuboot_buf = Util_amalloc(DFU_BUF_SZ);
	dfu_target_mcuboot_set_buf(mcuboot_buf, DFU_BUF_SZ);

	int ret = dfu_target_mcuboot_init(fsize, img_num, _dfu_cb);
	if (ret<0) {
		usb_fs_close_file(fh);
		Util_afree(buf);
		Util_afree(mcuboot_buf);
		log_warn("dfu : init for %s image identified as type mcuboot failed code %d", filename, ret);
		return false;
	}
	log_info("dfu: starting update of image for %s", filename);
	printk("Copying to secondard slot.");
	while(buf_sz==DFU_BUF_SZ) {
		// write the block
		int wret = dfu_target_mcuboot_write(buf, buf_sz);
		if (wret==0) {
			// read the next
			buf_sz = usb_fs_read_file(fh, buf, DFU_BUF_SZ);
			printk(".");
		} else {
			buf_sz = wret;
			printk("[WRITE FAIL %d]\n", wret);
		}
	}
	if (buf_sz>0) {
		// Write the final block
		buf_sz = dfu_target_mcuboot_write(buf, buf_sz);
		printk(". DONE\n");
	}
	usb_fs_close_file(fh);
	Util_afree(buf);
	Util_afree(mcuboot_buf);
	if (buf_sz<0) {
		log_warn("dfu : failed during install of file %s, %d", filename, buf_sz);
		dfu_target_mcuboot_done(false);
		// TBD : should we delete the 'bad' bin file? or try again next boot?
		return false;
	}
	// Success! Delete the file (we don't want it hanging about and triggering dfu again next boot time!)
	usb_fs_delete_file(filename);
	int dret = dfu_target_mcuboot_done(true);
	if (dret==0) {
		dret = dfu_target_mcuboot_schedule_update(img_num);
		if (dret==0) {
			return true;
		} else {
			log_warn("dfu : schedule_update() returns error %d, dfu filed for %s", dret, filename);
		}
	} else {
		log_warn("dfu : done() returns error %d, dfu filed for %s", dret, filename);
	}
	return false;
}

And my pm_static.yml also, as this is also a pain in the neck to get right...

mcuboot:
    address: 0x0
    end_address: 0xe000
    region: flash_primary
    size: 0xe000

mcuboot_pad:
    address: 0xe000
    end_address: 0xe200
    region: flash_primary
    size: 0x200

app:
    address: 0xe200
    end_address: 0x100000
    region: flash_primary
    size: 0xf1e00

mcuboot_primary_app:
    address: 0xe200
    end_address: 0x100000
    region: flash_primary
    size: 0xf1e00
    span: [app]

mcuboot_primary:
    address: 0xe000
    end_address: 0x100000
    region: flash_primary
    size: 0xf2000
    span: [mcuboot_pad, mcuboot_primary_app]

# define partition for fatfs disk, on the external flash storage (probably the device defined by nordic,pm-ext-flash in the DTS)
fatfs_storage:
    region: external_flash
    affiliation: 
        - disk
    extra_params: {
        disk_name: "NAND",
        disk_cache_size: 4096,
        disk_sector_size: 512,
        disk_read_only: 0
    }
    # 6Mb size, external flash is 8Mb total
    address: 0x0
    size: 0x600000

nvs_storage:
    region: external_flash
    affiliation: 
        - nvs
    # 64kB size, its just for small stuff
    address: 0x600000
    size: 0x10000

mcuboot_secondary_1:
    region: external_flash
    affiliation: 
        - mcuboot
    # 256kb size, its for netcore flash image
    address: 0x610000
    size: 0x40000

mcuboot_secondary:
    region: external_flash
    affiliation: 
        - mcuboot
    address: 0x650000
    # MUST be same size as the primary application slot size, and share_size does not appear to work
    size: 0xf1e00
    #share_size: [mcuboot_primary_app]

# internal cuisine
otp:
    address: 0xff8100
    end_address: 0xff83fc
    region: otp
    size: 0x2fc
pcd_sram:
    address: 0x20000000
    end_address: 0x20002000
    #placement:
    #    after:
    #    - start
    region: sram_primary
    size: 0x2000
rpmsg_nrf53_sram:
    address: 0x20070000
    end_address: 0x20080000
    #placement:
    #    before:
    #    - end
    region: sram_primary
    size: 0x10000
sram_primary:
    address: 0x20002000
    end_address: 0x20070000
    region: sram_primary
    size: 0x6e000

# TODO need fake flash ram flash partition to do cpunet DFU...
#mcuboot_primary_1:
#  address: 0x0
#  device: nordic_ram_flash_controller
#  end_address: 0x40000
#  region: ram_flash
#  size: 0x40000
#
#ram_flash:
#  address: 0x40000
#  end_address: 0x40000
#  region: ram_flash
#  size: 0x0

(you're welcome)

Note app_update.bin is the file generated by the build, and has the mcuboot magic number at the start...

I haven't yet tried the CPUNET update (as you can see the ram flash simulator is still commented out).

BrianW · Answer

So.... a CRITICAL thing I have discovered, which is not IMHO well documented, is that it is essential that your slot partitions for mcuboot (primary, secondary) are both aligned at their start address on a flash erase sector boundry AND that their size is ALSO aligned on a erase boundry (0x1000=4Kb for my flash device for example). 
 The 'mcuboot_pad' partition (for the 1st image ie the app) is the one to align on the boundry, not the following 'app' partition (so that the 'mcuboot_primary' partition ends up on the boundary). 
 Its the 'mcuboot_primary' size that must be aligned, and the 'mcuboot_secondary' must be this size, not the app size.... 
 If you don't align the start of the particion : after a mcuboot swap, the resulting primary image is unreadable for booting (presumably because it couldn't erase the start sector properly?) 
 If you don't align the size of the primary/secondary : the swap procedure fails to clear the mcuboot trailer (because its size is not aligned for erase?), and your 2 slots will flip-flop on reboot.... 
 When you are trying to use every last byte of the internal flash for your app (eg because the wifi stack is taking up 3/4 of the flash....) then the temptation is to avoid any wasted space... but the alignment needs to be there as the mcuboot code doesn't cope otherwise.... 
 Here's my new pm_static.yml : note the 'app' is not an aligned size, but both mcuboot_primary and mcuboot_secondary are... 
 mcuboot:
 address: 0x0
 end_address: 0x11000
 region: flash_primary
 affiliation: 
 - mcuboot
 size: 0x11000

mcuboot_pad:
 # should be aligned on CONFIG_FPROTECT_BLOCK_SIZE (0x4000 on nrf5340) boundry to allow FPROTECT to be enabled
 # and AT minimum on 0x1000 sector size for erase needs (otherwise DFU (move algo) will NOT WORK)
 address: 0x11000
 end_address: 0x11200
 region: flash_primary
 affiliation: 
 - mcuboot
 size: 0x200

app:
 address: 0x11200
 end_address: 0x100000
 region: flash_primary
 size: 0xeee00

mcuboot_primary_app:
 address: 0x11200
 end_address: 0x100000
 region: flash_primary
 affiliation: 
 - mcuboot
 size: 0xeee00
 span: [app]

mcuboot_primary:
 address: 0x11000
 end_address: 0x100000
 region: flash_primary
 affiliation: 
 - mcuboot
 size: 0xef000
 span: [mcuboot_pad, app]

# define partition for fatfs disk, on the external flash storage (probably the device defined by nordic,pm-ext-flash in the DTS)
fatfs_storage:
 region: external_flash
 affiliation: 
 - disk
 extra_params: {
 disk_name: "NAND",
 disk_cache_size: 4096,
 disk_sector_size: 512,
 disk_read_only: 0
 }
 # 6Mb size, external flash is 8Mb total
 address: 0x0
 size: 0x600000

nvs_storage:
 region: external_flash
 affiliation: 
 - nvs
 # 64kB size, its just for small stuff
 address: 0x600000
 size: 0x10000

mcuboot_secondary_1:
 region: external_flash
 affiliation: 
 - mcuboot
 # 256kb size, its for netcore flash image
 address: 0x610000
 size: 0x40000

mcuboot_secondary:
 region: external_flash
 affiliation: 
 - mcuboot
 address: 0x650000
 # MUST be same size as the 'mcuboot_primary' slot size, and share_size does not appear to work
 size: 0xef000
 #share_size: [mcuboot_primary]

# get next slot to be aligned on 0x4000
PAD1_END:
 region: external_flash
 address: 0x73f000
 size: 0x1000

# allow wifi patches to be updated by mcuboot
nrf70_wifi_fw_mcuboot_pad:
 region: external_flash
 address: 0x740000
 size: 0x200
 #device: MX25R64

nrf70_wifi_fw:
 region: external_flash
 # 128kB size for the wifi fw (-0x200 bytes, the mcuboot pad so that total is aligned on 0x1000 - and 0x4000 - to keep mcuboot happy), 
 address: 0x740200
 size: 0x1fe00

mcuboot_primary_2:
 region: external_flash
 orig_span: &id003
 - nrf70_wifi_fw_mcuboot_pad
 - nrf70_wifi_fw
 span: *id003
 address: 0x740000
 affiliation: 
 - mcuboot
 size: 0x20000
 #device: MX25R64

mcuboot_secondary_2:
 region: external_flash
 address: 0x760000
 affiliation: 
 - mcuboot
 size: 0x20000
 #device: MX25R64

# remaining external flash space starts at 0x78000

# need fake flash ram flash partition to do cpunet DFU...
mcuboot_primary_1:
 region: ram_flash
 address: 0x0
 affiliation: 
 - mcuboot
 size: 0x40000
 device: flash_ctrl
 # end_address: 0x40000
 # device: nordic_ram_flash_controller

ram_flash:
 address: 0x40000
 end_address: 0x40000
 region: ram_flash
 size: 0x0

# internal cuisine
#otp:
# address: 0xff8100
# end_address: 0xff83fc
# region: otp
# size: 0x2fc
#pcd_sram:
# address: 0x20000000
# end_address: 0x20002000
# region: sram_primary
# size: 0x2000
#rpmsg_nrf53_sram:
# address: 0x20070000
# end_address: 0x20080000
# region: sram_primary
# size: 0x10000
#sram_primary:
# address: 0x20002000
# end_address: 0x20070000
# region: sram_primary
# size: 0x6e000

 
 And with this layout that critically aligned the size of the mcuboot_secondary on a flash erase boundary , the mcuboot 'swap-using-move' procedure plus the call to boot_write_img_confirmed_multi () mean that the image no longer flip flops on each boot! 
 This should really be in the mcuboot/bootloader doc as its a critical point and not obvious.... 
 by the way, it would be nice if in the dfu_target subsystem dfu_target_mcuboot.c provided the required multi-image confirmation functions 
 dfu_target_mcuboot_is_img_confirmed_multi(int img_num) 
 dfu_target_mcuboot_img_confirm_multi(int img_num)