AWS FOTA Failure on nRF9160

[Craig12]

We have deployed a fleet of nRF9160 based devices communicating with AWS. The production code currently deployed is based on SDK 1.9.1.

Previously we have pushed out firmware and modem FOTA updates without issue using AWS.

We have now upgraded to nRFConnect SDK 2.2.0. I seem to be able to push an upgrade from our old code base built upon 1.9.1 to 2.2.0. However, once we have upgraded to 2.2.0 all subsequent app firmware updates seem to fail.

I was hoping on some general guidance on where to look.

A successful upgrade from an app on 1.9.1 (our production code) to an app on 2.2.0 looks like the following: (I note MCUboot indicates the secondary image magic is good.)

I: Downloaded 219628/219628 bytes (100%)
I: Download complete
I: MCUBoot image upgrade scheduled. Reset device to apply
AWS_IOT_EVT_FOTA_DL_PROGRESS, (100%)AWS_IOT_EVT_ERROR, 100
AWS_IOT_EVT_FOTA_DONE
FOTA done, rebooting device
AWS_IOT_EVT_DISCONNECTED
W: connection_poll_start failed, error: -119
aws_iot_connect, error: -119
Next connection retry in 30 seconds
þ*** Booting Zephyr OS build v2.7.99-ncs1-1  ***
I: Starting bootloader
I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Swap type: test
I: Bootloader chainload address offset: 0x10000
I: Jumping to the first image slot
*** Booting Zephyr OS build v3.2.99-ncs1 ***

If we then perform an update from the above app to a new app also built on 2.2.0 we get the following (this time the magic is unset?)

I: Downloading: app_update_222.bin [0]
...
I: Downloaded 219628/219628 bytes (100%)
I: Download complete
I: MCUBoot image-0 upgrade scheduled. Reset device to apply
AWS_IOT_EVT_FOTA_DL_PROGRESS, (100%)AWS_IOT_EVT_ERROR, 100
AWS_IOT_EVT_FOTA_DONE
FOTA done, rebooting device
AWS_IOT_EVT_DISCONNECTED
þ*** Booting Zephyr OS build v2.7.99-ncs1-1  ***
I: Starting bootloader
I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Swap type: none
I: Bootloader chainload address offset: 0x10000
I: Jumping to the first image slot
*** Booting Zephyr OS build v3.2.99-ncs1 ***

To verify the above image is actually good, we can via SWD load an app+mcuboot based on SDK 2.2.0. In this case (with the upgraded MCUboot, it is happy to take the image and reports the magic is good)

I: Downloading: app_update_222.bin [0]
...
I: Downloaded 219628/219628 bytes (100%)
I: Download complete
I: MCUBoot image-0 upgrade scheduled. Reset device to apply
AWS_IOT_EVT_FOTA_DL_PROGRESS, (100%)AWS_IOT_EVT_ERROR, 100
Unknown AWS IoT event type: 6
AWS_IOT_EVT_FOTA_DONE
FOTA done, rebooting device
AWS_IOT_EVT_DISCONNECTED
þ*** Booting Zephyr OS build v3.2.99-ncs1 ***
I: Starting bootloader
I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Secondary image: magic=good, swap_type=0x2, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Swap type: test
I: Starting swap using move algorithm.
I: Bootloader chainload address offset: 0x10000
I: Jumping to the first image slot
*** Booting Zephyr OS build v3.2.99-ncs1 ***

To migrate to 2.2.0 we have had to upgrade board files. I have looked at the flash memory map for slot0 and slot1 and they appear to be the same. I was thinking the later app might be writing the new firmware to a different location to what the old MCUboot was expecting. The new MCUboot, built at the same time than the app, probably has the same slot address.

I assume the app writes the new firmware to slot1 and then via some mechanism flags MCUboot to swap at reboot.

[Craig12]

I just had a look at the partitions.yml left behind in the build directory from the two projects.

The project built with SDK 1.8.1 has

mcuboot_secondary:
  address: 0x88000

SDK 2.2.0 has
mcuboot_secondary:
  address: 0x80000

I think I'll do a little more digging on flash partitions.

[Øyvind]

Hello, 

Have you remembered to configure a static configuration of your project? This ensures that the flash partitions stays the same between upgrades using a pm_static.

The current partition configuration for a build can be found in $BUILD_DIR/partitions.yml. To apply the current configuration as a static configuration, copy this file to $APPLICATION_CONFIG_DIR/pm_static.yml.

Kind regards,
Øyvind

[Craig12]

Thanks Oyvind. That makes perfect sense.

It has got us closer, but I suspect we now have a memory map allocation issue we have to work through.

If I copied the partitions.yml over verbatim to pm_static.yml, it failed to build with "Shared data area and non-secure data area is overlapping"

So instead, I statically assigned just the mcuboot_secondary address and size. I could now get FOTA to work and mcuboot to pick up the image and swap them. So it verifies the change of mcuboot_secondary address was indeed the problem.

However, it then got in a loop and would reboot and download the image again with MCUboot sometimes doing a revert.

*** Booting Zephyr OS build v2.7.99-ncs1-1  ***
I: Starting bootloader
I: Primary image: magic=good, swap_type=0x2, copy_done=0x1, image_ok=0x3
I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Swap type: revert
I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Bootloader chainload address offset: 0x10000
I: Jumping to the first image slot
*** Booting Zephyr OS build v3.2.99-ncs1 ***

Looking at MCUboot documentation it appears there is an image trailer at the end of each slot to determine the current state. So I suspect the mcuboot_primary end_address may in incorrect as well and the image is never being set as OK.

I'll continue to work through the problem for now. Thanks for your help.