nRF5340: MCUBoot issue with multi image update when SB_CONFIG_SECURE_BOOT_APPCORE is enabled

[Bogdan Marinescu]

Hi. I had a working MCUboot configuration that I used in order to update both the app core and the net core as follows:

- Upload the app core image using mcumgr
- Upload the net core image using mcumgr
- Activate the app core image using mcumgr (https://docs.zephyrproject.org/latest/services/device_mgmt/smp_groups/smp_group_1.html#set-state-of-image-request)
- Activate the net core image using mcumgr
- Reboot

This was with NCS 2.6.1 and without sysbuild. I recently switched to sysbuild (and 2.7.0) and everything still works as long as I don't enable SB_CONFIG_SECURE_BOOT_APPCORE (app core secure boot was also disabled in my previous, pre-sysbuild configuration). In other words, everything still works with this sysbuild.conf:

SB_CONFIG_NETCORE_HCI_IPC=y

SB_CONFIG_BOOTLOADER_MCUBOOT=y

SB_CONFIG_DFU_MULTI_IMAGE_PACKAGE_BUILD=y

SB_CONFIG_DFU_MULTI_IMAGE_PACKAGE_APP=y

SB_CONFIG_DFU_MULTI_IMAGE_PACKAGE_NET=y

SB_CONFIG_PARTITION_MANAGER=y

SB_CONFIG_PM_EXTERNAL_FLASH_MCUBOOT_SECONDARY=y

SB_CONFIG_BUILD_OUTPUT_BIN=y

SB_CONFIG_BUILD_OUTPUT_HEX=y

SB_CONFIG_MCUBOOT_UPDATEABLE_IMAGES=2

SB_CONFIG_SECURE_BOOT_APPCORE=n

SB_CONFIG_SECURE_BOOT_NETCORE=y

SB_CONFIG_MCUBOOT_APP_SYNC_UPDATEABLE_IMAGES=y

SB_CONFIG_MCUBOOT_MODE_OVERWRITE_ONLY=y

SB_CONFIG_NETCORE_APP_UPDATE=y

In this working configuration, after I upload and activate the firmware images as described above, MCUboot updates both (emphasis mine):

*** Booting MCUboot v2.1.0-dev-5f22c1164620 ***
*** Using nRF Connect SDK v2.7.0-9135e9629c0a ***
*** Using Zephyr OS v3.6.99-ba7fbf2a8ec5 ***
I: Starting bootloader
I: Image index: 0, Swap type: test
I: Image index: 1, Swap type: test
I: Image 0 upgrade secondary slot -> primary slot
I: Erasing the primary slot
I: Image 0 copying the secondary slot to the primary slot: 0x76710 bytes
I: Image 1 upgrade secondary slot -> primary slot
I: Erasing the primary slot
I: Image 1 copying the secondary slot to the primary slot: 0x30210 bytes
I: Turned on network core
I: Turned off network core
I: Bootloader chainload address offset: 0x14000

This is the paritition manager report in this working configuration:

external_flash (0x800000 - 8192kB):
+-------------------------------------------------+
| 0x0: littlefs_storage (0x100000 - 1024kB) |
| 0x100000: mcuboot_secondary (0xec000 - 944kB) |
| 0x1ec000: mcuboot_secondary_1 (0x40000 - 256kB) |
| 0x22c000: external_flash (0x5d4000 - 5968kB) |
+-------------------------------------------------+

flash_primary (0x100000 - 1024kB):
+--------------------------------------------------+
| 0x0: mcuboot (0x12800 - 74kB) |
| 0x12800: EMPTY_0 (0x1800 - 6kB) |
+---0x14000: mcuboot_primary (0xec000 - 944kB)-----+
| 0x14000: mcuboot_pad (0x200 - 512B) |
+---0x14200: mcuboot_primary_app (0xebe00 - 943kB)-+
| 0x14200: app (0xebe00 - 943kB) |
+--------------------------------------------------+

otp (0x2fc - 764B):
+------------------------------+
| 0xff8100: otp (0x2fc - 764B) |
+------------------------------+

ram_flash (0x40000 - 256kB):
+------------------------------------------+
| 0x0: mcuboot_primary_1 (0x40000 - 256kB) |
| 0x40000: ram_flash (0x0 - 0B) |
+------------------------------------------+

sram_primary (0x80000 - 512kB):
+--------------------------------------------+
| 0x20000000: pcd_sram (0x2000 - 8kB) |
| 0x20002000: sram_primary (0x6e000 - 440kB) |
| 0x20070000: sram_shared0 (0xf000 - 60kB) |
| 0x2007f000: sram_shared1 (0x1000 - 4kB) |
+--------------------------------------------+

CPUNET flash_primary (0x40000 - 256kB):
+--------------------------------------------+
+---0x1000000: b0n_container (0x4000 - 16kB)-+
| 0x1000000: b0n (0x3d80 - 15kB) |
| 0x1003d80: provision (0x280 - 640B) |
+---0x1004000: app (0x3c000 - 240kB)---------+
| 0x1004000: hci_ipc (0x3c000 - 240kB) |
+--------------------------------------------+

CPUNET sram_primary (0x10000 - 64kB):
+-------------------------------------------+
| 0x21000000: sram_primary (0x10000 - 64kB) |
+-------------------------------------------+

This changes as soon as I set SB_CONFIG_SECURE_BOOT_APPCORE=y in sysbuild.conf. Now, if I run the exact same upgrade sequence as before, this is what I get from MCUboot (emphasis mine):

*** Booting MCUboot v2.1.0-dev-5f22c1164620 ***
*** Using nRF Connect SDK v2.7.0-9135e9629c0a ***
*** Using Zephyr OS v3.6.99-ba7fbf2a8ec5 ***
I: Starting bootloader
I: Image index: 0, Swap type: test
I: Image index: 1, Swap type: none
I: Image 0 upgrade secondary slot -> primary slot
I: Erasing the primary slot
I: Image 0 copying the secondary slot to the primary slot: 0x76698 bytes
I: Bootloader chainload address offset: 0x30000

This time, MCUboot doesn't recognize the swap type as "test" on image index 1 (net core image) and thus it doesn't try to update it (although, again, the way the firmware images are updated and activated is the same). Interestingly, if I try to update just the network core using the same method, it does work. The only thing that doesn't work is this combined app core + net core update.
This is the partition manager report in this partially working configuration:

external_flash (0x800000 - 8192kB):
+-------------------------------------------------+
| 0x0: littlefs_storage (0x100000 - 1024kB) |
| 0x100000: mcuboot_secondary (0xd0000 - 832kB) |
| 0x1d0000: mcuboot_secondary_1 (0x40000 - 256kB) |
| 0x210000: external_flash (0x5f0000 - 6080kB) |
+-------------------------------------------------+

flash_primary (0x100000 - 1024kB):
+--------------------------------------------------+
+---0x0: b0_container (0x8000 - 32kB)--------------+
| 0x0: b0 (0x8000 - 32kB) |
+---0x8000: s0 (0x12a00 - 74kB)--------------------+
| 0x8000: s0_pad (0x200 - 512B) |
+---0x8200: s0_image (0x12800 - 74kB)--------------+
| 0x8200: mcuboot (0x12800 - 74kB) |
+--------------------------------------------------+
| 0x1aa00: EMPTY_0 (0x1600 - 5kB) |
+---0x1c000: s1 (0x12a00 - 74kB)-------------------+
| 0x1c000: s1_pad (0x200 - 512B) |
| 0x1c200: s1_image (0x12800 - 74kB) |
+--------------------------------------------------+
| 0x2ea00: EMPTY_1 (0x1600 - 5kB) |
+---0x30000: mcuboot_primary (0xd0000 - 832kB)-----+
| 0x30000: mcuboot_pad (0x200 - 512B) |
+---0x30200: app_image (0xcfe00 - 831kB)-----------+
+---0x30200: mcuboot_primary_app (0xcfe00 - 831kB)-+
| 0x30200: app (0xcfe00 - 831kB) |
+--------------------------------------------------+

otp (0x2fc - 764B):
+------------------------------------+
| 0xff8100: provision (0x280 - 640B) |
| 0xff8380: otp (0x7c - 124B) |
+------------------------------------+

ram_flash (0x40000 - 256kB):
+------------------------------------------+
| 0x0: mcuboot_primary_1 (0x40000 - 256kB) |
| 0x40000: ram_flash (0x0 - 0B) |
+------------------------------------------+

sram_primary (0x80000 - 512kB):
+--------------------------------------------+
| 0x20000000: pcd_sram (0x2000 - 8kB) |
| 0x20002000: sram_primary (0x6e000 - 440kB) |
| 0x20070000: sram_shared0 (0xf000 - 60kB) |
| 0x2007f000: sram_shared1 (0x1000 - 4kB) |
+--------------------------------------------+

CPUNET flash_primary (0x40000 - 256kB):
+--------------------------------------------+
+---0x1000000: b0n_container (0x4000 - 16kB)-+
| 0x1000000: b0n (0x3d80 - 15kB) |
| 0x1003d80: provision (0x280 - 640B) |
+---0x1004000: app (0x3c000 - 240kB)---------+
| 0x1004000: hci_ipc (0x3c000 - 240kB) |
+--------------------------------------------+

CPUNET sram_primary (0x10000 - 64kB):
+-------------------------------------------+
| 0x21000000: sram_primary (0x10000 - 64kB) |
+-------------------------------------------+

I'm running out of ideas on this, so please let me know if you can help. Thanks.

[Bogdan Marinescu]

I found a fix for this, although probably not the best one. Turns out that changing the update sequence from:

- Upload the app core image using mcumgr
- Upload the net core image using mcumgr
- Activate the app core image using mcumgr
- Activate the net core image using mcumgr
- Reboot

to:

- Upload the net core image using mcumgr
- Upload the app core image using mcumgr
- Activate the net core image using mcumgr
- Activate the app core image using mcumgr
- Reboot

fixes the problem and MCUboot updates both the app core and the net core once again, even with SB_CONFIG_SECURE_BOOT_APPCORE=y. Pretty sure that this "fix" only masks the main issue, but for now it does the job.

[AHaug]

I'm not sure why you're having issues when doing it the initial way, but if your solution works then it's a proper fix.

Let me know if you find any issues with this method after a while and we'll look closer into it.

Kind regards,
Andreas

[AHaug]

Hi,

Alexander let me know you had commented here

Hayden Ball said:

This feels like there is a deeper problem, as Bogdan Marinescu has mentioned, and this solution masks the actual problem.

There still could be a problem indeed. I see the reviewers are on it and I've reached out to them and asked for some input and shared the context of this case with them. If it becomes relevant, I will relay any info from them to you through here

Kind regards,
Andreas

[Hayden Ball]

Thanks AHaug 

If the team have any feedback I'd really appreciate it - this is blocking our product beta launch.

[AHaug]

Hi,

Thanks for letting me know. I've let the developers know about the urgency

Kind regards,
Andreas

[AHaug]

Initial comment (without testing): "It looks like the PR is overall ok, except for the lack of nrf tag"

Kind regards,
Andreas

[AHaug]

Hi again,

Here's an update from internally. There's a workaround that you can use below. W.r.t for a PR for a fix for this issue, I'm afraid it will take some time, specially since the workaround seems to work. Please report back if the suggestion does not work as intended.

Changing the update sequence from:

• Upload the app core image using mcumgr
• Upload the net core image using mcumgr
• Activate the app core image using mcumgr
• Activate the net core image using mcumgr
• Reboot

to:

• Upload the net core image using mcumgr
• Upload the app core image using mcumgr
• Activate the net core image using mcumgr
• Activate the app core image using mcumgr
• Reboot

fixes the problem and MCUboot updates both the app core and the net core once again, even with SB_CONFIG_SECURE_BOOT_APPCORE=y.

Kind regards,
Andreas