NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

Hung Bui — Mon, 07 Oct 2019 09:03:25 GMT

Great to know GJSea. Thanks for the update :)

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

GJSea — Mon, 07 Oct 2019 06:02:09 GMT

Hi Hung,

I figured the problem out, I was using NVS in our firmware app without changing nvs_fs.offset to PM_MCUBOOT_STORAGE_ADDRESS which was somehow corrupting the written secondary firmware. Now it it working nicely.

Thanks for your help in aiding me narrowing it down.

Thx!

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

Hung Bui — Fri, 04 Oct 2019 08:55:22 GMT

Hi,

I'm a bit out of idea on what could be wrong. We may need to have your code to reproduce the issue here. The problem is that we may not have the SPI external flash as you have in your setup.

If you can modify the code to copy from internal flash to slot 2 so we can test on a standalone board it would be great.

One thing you can do also is to read the hex dump and check if the update.bin image matched with what copied to slot 2. I understand that you have the CRC check but it could be offset. If possible can you send us the hex dump right after the FOTA ?

Regarding the check inside boot_validate_slot() here is the explanation from the team:

> if (boot_check_header_erased(slot) == 0 || (hdr->ih_flags & IMAGE_F_NON_BOOTABLE)) {

Check if the magic field in image header is erased or non-bootable flag is set – this will be satisfied if for instance header structure is erased (this is my suspicion). Looks like what is written to the image slot is not the same as in update.bin, or with a offset.

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

GJSea — Fri, 04 Oct 2019 03:04:50 GMT

Hi Hung,

Yes CONFIG_IMG_ERASE_PROGRESSIVELY was defined properly, I also added boot_erase_img_bank for good measure. However after that MCU_BOOT won't even load from the PRIMARY slot:

bootutil_img_validate() fails in comparing the hash at

if (memcmp(hash, buf, sizeof(hash))) {
return -1;
}

Segger properly downloads the firmware image to the device as clearly I'm debugging MCUBOOT, can you provide guidance on how to restore booting the PRIMARY slot, so we can continue further in diagnosis?

Thanks!

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

Hung Bui — Thu, 03 Oct 2019 12:04:12 GMT

Hi again GJSea,

When doing our investigation we came across this case : https://github.com/zephyrproject-rtos/zephyr/issues/19438

It looks quite similar to your issue. But, I don't think t's the same. The case was about booting via the image on QSPI when in your case it's the internal slot .

But it could be related, we need to have a look at how you erase slot 2 prior to the swap.

Have you defined CONFIG_IMG_ERASE_PROGRESSIVELY = y (or SELECT it )

It's selected inside fota_download\Kconfig file in our http applicatoin update example .

If CONFIG_IMG_ERASE_PROGRESSIVELY is not selected, you would need to do a boot_erase_img_bank() before flash_img_init()

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

Hung Bui — Wed, 02 Oct 2019 11:09:21 GMT

Hi GJSea,

I'm sorry for late response. I have forwarded the case to our MCUBoot team and will get back to you asap.

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

GJSea — Tue, 01 Oct 2019 03:50:27 GMT

Hi,

So I did all three things above to no avail (CONFIG_IMG_BLOCK_BUF_SIZE, checking flash_img.bytes_written, and the small delay).

I ended up debugging mcuboot\boot\bootutil\src\LOADER.C to diagnose this further:

In boot_validate_slot(...) boot_check_header_erased is the function that is failing.

flash_area_erased_val() returns 0xFF, as is comparing it to hdr->ih_magic (IMAGE_MAGIC) which contains values other than 0xFF, so returning -1 hence boot_validated_swap_type eventually loads the PRIMARY image again.

So why is the ih_magic number from the SECONDARY's boot_read_image_header() not returning 0xFF?

Thanks

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

Hung Bui — Wed, 25 Sep 2019 13:34:33 GMT

Also, could you try adding a small delay before triggering the reset ? (1 -2 seconds for example)

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

Hung Bui — Wed, 25 Sep 2019 13:14:21 GMT

Hi GJSea,

It looks fine to me. Could you post the UART logging when you update and the device get reset?

I talked to the team and got some suggestion:

- To use CONFIG_IMG_BLOCK_BUF_SIZE instead of DOWNLOAD_MAX_FRAGMENT_SIZE when calculating readAmount

- Before triggering the reset, could you check flash_img.bytes_written if it matches with the image size ?

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

GJSea — Wed, 25 Sep 2019 05:09:30 GMT

Hi,

Yes the code was from fota_download, except we read the firmware bits from a file on SPI flash storage that we previously loaded over http (so we can cater for interrupted OTA byte range streaming). This has all been CRC32 checked so the integrity of the firmware binary stored in SPI flash that we're doing f_reads from is correct. I stripped the error checking out from the paste of the code below to help in legibility.

Yes we have the boot_write_img_confirmed at the beginning of our main.

int err = flash_img_init(&flash_img);

bytesRemaining = filesize;
while(bytesRemaining > 0)
{
	int readAmount = MIN(bytesRemaining, DOWNLOAD_MAX_FRAGMENT_SIZE);

    fileResult = f_read(fwFile, recv_buf, readAmount, &dr);       
    err = flash_img_buffered_write(&flash_img, (uint8_t *) recv_buf, readAmount, false);
    bytesRemaining -= readAmount;
}

// Write with 0 length to flush the write operation to flash. 
flash_img_buffered_write(&flash_img,(uint8_t *)recv_buf, 0, true);

boot_request_upgrade(BOOT_UPGRADE_TEST);

// Reboot system to apply the firmware.
sys_reboot(SYS_REBOOT_COLD);

Thx!

RE: NRF9160 FOTA not switching to secondary after boot_request_upgrade() and sys_reboot()

Hung Bui — Tue, 24 Sep 2019 14:15:40 GMT

Hi GJSea,

Could you provide us some more of your code where you handle FOTA ? I assume you are writing your own code based on our libraries, could you list the libraries you are using ?

Please make sure that you have boot_write_img_confirmed() in your new firmware as showed in our example. Otherwise the next reset it will return back to the Primary image.

When you write using flash_img_buffered_write() do you flush the flash ? that what we do at:

https://github.com/NordicPlayground/fw-nrfconnect-nrf/blob/b456fb40c4912e3d435b9ee029d8ef24b113f7d0/subsys/net/lib/fota_download/src/fota_download.c#L59

You need to do that before resetting.

I assume you have tested with our http application update and it worked fine ?