nRF5340: MCUBoot issue with multi image update when SB_CONFIG_SECURE_BOOT_APPCORE is enabled

[Bogdan Marinescu]

Hi. I had a working MCUboot configuration that I used in order to update both the app core and the net core as follows:

- Upload the app core image using mcumgr
- Upload the net core image using mcumgr
- Activate the app core image using mcumgr (https://docs.zephyrproject.org/latest/services/device_mgmt/smp_groups/smp_group_1.html#set-state-of-image-request)
- Activate the net core image using mcumgr
- Reboot

This was with NCS 2.6.1 and without sysbuild. I recently switched to sysbuild (and 2.7.0) and everything still works as long as I don't enable SB_CONFIG_SECURE_BOOT_APPCORE (app core secure boot was also disabled in my previous, pre-sysbuild configuration). In other words, everything still works with this sysbuild.conf:

SB_CONFIG_NETCORE_HCI_IPC=y

SB_CONFIG_BOOTLOADER_MCUBOOT=y

SB_CONFIG_DFU_MULTI_IMAGE_PACKAGE_BUILD=y

SB_CONFIG_DFU_MULTI_IMAGE_PACKAGE_APP=y

SB_CONFIG_DFU_MULTI_IMAGE_PACKAGE_NET=y

SB_CONFIG_PARTITION_MANAGER=y

SB_CONFIG_PM_EXTERNAL_FLASH_MCUBOOT_SECONDARY=y

SB_CONFIG_BUILD_OUTPUT_BIN=y

SB_CONFIG_BUILD_OUTPUT_HEX=y

SB_CONFIG_MCUBOOT_UPDATEABLE_IMAGES=2

SB_CONFIG_SECURE_BOOT_APPCORE=n

SB_CONFIG_SECURE_BOOT_NETCORE=y

SB_CONFIG_MCUBOOT_APP_SYNC_UPDATEABLE_IMAGES=y

SB_CONFIG_MCUBOOT_MODE_OVERWRITE_ONLY=y

SB_CONFIG_NETCORE_APP_UPDATE=y

In this working configuration, after I upload and activate the firmware images as described above, MCUboot updates both (emphasis mine):

*** Booting MCUboot v2.1.0-dev-5f22c1164620 ***
*** Using nRF Connect SDK v2.7.0-9135e9629c0a ***
*** Using Zephyr OS v3.6.99-ba7fbf2a8ec5 ***
I: Starting bootloader
I: Image index: 0, Swap type: test
I: Image index: 1, Swap type: test
I: Image 0 upgrade secondary slot -> primary slot
I: Erasing the primary slot
I: Image 0 copying the secondary slot to the primary slot: 0x76710 bytes
I: Image 1 upgrade secondary slot -> primary slot
I: Erasing the primary slot
I: Image 1 copying the secondary slot to the primary slot: 0x30210 bytes
I: Turned on network core
I: Turned off network core
I: Bootloader chainload address offset: 0x14000

This is the paritition manager report in this working configuration:

external_flash (0x800000 - 8192kB):
+-------------------------------------------------+
| 0x0: littlefs_storage (0x100000 - 1024kB) |
| 0x100000: mcuboot_secondary (0xec000 - 944kB) |
| 0x1ec000: mcuboot_secondary_1 (0x40000 - 256kB) |
| 0x22c000: external_flash (0x5d4000 - 5968kB) |
+-------------------------------------------------+

flash_primary (0x100000 - 1024kB):
+--------------------------------------------------+
| 0x0: mcuboot (0x12800 - 74kB) |
| 0x12800: EMPTY_0 (0x1800 - 6kB) |
+---0x14000: mcuboot_primary (0xec000 - 944kB)-----+
| 0x14000: mcuboot_pad (0x200 - 512B) |
+---0x14200: mcuboot_primary_app (0xebe00 - 943kB)-+
| 0x14200: app (0xebe00 - 943kB) |
+--------------------------------------------------+

otp (0x2fc - 764B):
+------------------------------+
| 0xff8100: otp (0x2fc - 764B) |
+------------------------------+

ram_flash (0x40000 - 256kB):
+------------------------------------------+
| 0x0: mcuboot_primary_1 (0x40000 - 256kB) |
| 0x40000: ram_flash (0x0 - 0B) |
+------------------------------------------+

sram_primary (0x80000 - 512kB):
+--------------------------------------------+
| 0x20000000: pcd_sram (0x2000 - 8kB) |
| 0x20002000: sram_primary (0x6e000 - 440kB) |
| 0x20070000: sram_shared0 (0xf000 - 60kB) |
| 0x2007f000: sram_shared1 (0x1000 - 4kB) |
+--------------------------------------------+

CPUNET flash_primary (0x40000 - 256kB):
+--------------------------------------------+
+---0x1000000: b0n_container (0x4000 - 16kB)-+
| 0x1000000: b0n (0x3d80 - 15kB) |
| 0x1003d80: provision (0x280 - 640B) |
+---0x1004000: app (0x3c000 - 240kB)---------+
| 0x1004000: hci_ipc (0x3c000 - 240kB) |
+--------------------------------------------+

CPUNET sram_primary (0x10000 - 64kB):
+-------------------------------------------+
| 0x21000000: sram_primary (0x10000 - 64kB) |
+-------------------------------------------+

This changes as soon as I set SB_CONFIG_SECURE_BOOT_APPCORE=y in sysbuild.conf. Now, if I run the exact same upgrade sequence as before, this is what I get from MCUboot (emphasis mine):

*** Booting MCUboot v2.1.0-dev-5f22c1164620 ***
*** Using nRF Connect SDK v2.7.0-9135e9629c0a ***
*** Using Zephyr OS v3.6.99-ba7fbf2a8ec5 ***
I: Starting bootloader
I: Image index: 0, Swap type: test
I: Image index: 1, Swap type: none
I: Image 0 upgrade secondary slot -> primary slot
I: Erasing the primary slot
I: Image 0 copying the secondary slot to the primary slot: 0x76698 bytes
I: Bootloader chainload address offset: 0x30000

This time, MCUboot doesn't recognize the swap type as "test" on image index 1 (net core image) and thus it doesn't try to update it (although, again, the way the firmware images are updated and activated is the same). Interestingly, if I try to update just the network core using the same method, it does work. The only thing that doesn't work is this combined app core + net core update.
This is the partition manager report in this partially working configuration:

external_flash (0x800000 - 8192kB):
+-------------------------------------------------+
| 0x0: littlefs_storage (0x100000 - 1024kB) |
| 0x100000: mcuboot_secondary (0xd0000 - 832kB) |
| 0x1d0000: mcuboot_secondary_1 (0x40000 - 256kB) |
| 0x210000: external_flash (0x5f0000 - 6080kB) |
+-------------------------------------------------+

flash_primary (0x100000 - 1024kB):
+--------------------------------------------------+
+---0x0: b0_container (0x8000 - 32kB)--------------+
| 0x0: b0 (0x8000 - 32kB) |
+---0x8000: s0 (0x12a00 - 74kB)--------------------+
| 0x8000: s0_pad (0x200 - 512B) |
+---0x8200: s0_image (0x12800 - 74kB)--------------+
| 0x8200: mcuboot (0x12800 - 74kB) |
+--------------------------------------------------+
| 0x1aa00: EMPTY_0 (0x1600 - 5kB) |
+---0x1c000: s1 (0x12a00 - 74kB)-------------------+
| 0x1c000: s1_pad (0x200 - 512B) |
| 0x1c200: s1_image (0x12800 - 74kB) |
+--------------------------------------------------+
| 0x2ea00: EMPTY_1 (0x1600 - 5kB) |
+---0x30000: mcuboot_primary (0xd0000 - 832kB)-----+
| 0x30000: mcuboot_pad (0x200 - 512B) |
+---0x30200: app_image (0xcfe00 - 831kB)-----------+
+---0x30200: mcuboot_primary_app (0xcfe00 - 831kB)-+
| 0x30200: app (0xcfe00 - 831kB) |
+--------------------------------------------------+

otp (0x2fc - 764B):
+------------------------------------+
| 0xff8100: provision (0x280 - 640B) |
| 0xff8380: otp (0x7c - 124B) |
+------------------------------------+

ram_flash (0x40000 - 256kB):
+------------------------------------------+
| 0x0: mcuboot_primary_1 (0x40000 - 256kB) |
| 0x40000: ram_flash (0x0 - 0B) |
+------------------------------------------+

sram_primary (0x80000 - 512kB):
+--------------------------------------------+
| 0x20000000: pcd_sram (0x2000 - 8kB) |
| 0x20002000: sram_primary (0x6e000 - 440kB) |
| 0x20070000: sram_shared0 (0xf000 - 60kB) |
| 0x2007f000: sram_shared1 (0x1000 - 4kB) |
+--------------------------------------------+

CPUNET flash_primary (0x40000 - 256kB):
+--------------------------------------------+
+---0x1000000: b0n_container (0x4000 - 16kB)-+
| 0x1000000: b0n (0x3d80 - 15kB) |
| 0x1003d80: provision (0x280 - 640B) |
+---0x1004000: app (0x3c000 - 240kB)---------+
| 0x1004000: hci_ipc (0x3c000 - 240kB) |
+--------------------------------------------+

CPUNET sram_primary (0x10000 - 64kB):
+-------------------------------------------+
| 0x21000000: sram_primary (0x10000 - 64kB) |
+-------------------------------------------+

I'm running out of ideas on this, so please let me know if you can help. Thanks.

[Bogdan Marinescu]

I found a fix for this, although probably not the best one. Turns out that changing the update sequence from:

- Upload the app core image using mcumgr
- Upload the net core image using mcumgr
- Activate the app core image using mcumgr
- Activate the net core image using mcumgr
- Reboot

to:

- Upload the net core image using mcumgr
- Upload the app core image using mcumgr
- Activate the net core image using mcumgr
- Activate the app core image using mcumgr
- Reboot

fixes the problem and MCUboot updates both the app core and the net core once again, even with SB_CONFIG_SECURE_BOOT_APPCORE=y. Pretty sure that this "fix" only masks the main issue, but for now it does the job.

[AHaug]

I'm not sure why you're having issues when doing it the initial way, but if your solution works then it's a proper fix.

Let me know if you find any issues with this method after a while and we'll look closer into it.

Kind regards,
Andreas

[Hayden Ball]

AHaug  please can you advise how to achieve this using https://github.com/NordicSemiconductor/Android-nRF-Connect-Device-Manager and https://github.com/NordicSemiconductor/IOS-nRF-Connect-Device-Manager?

I'm seeing the same problem, and both of the above always upload the app core image first.

This feels like there is a deeper problem, as Bogdan Marinescu has mentioned, and this solution masks the actual problem.

Is there a way to fix this in firmware such that the "normal" order of app core, net core can be followed?

[Hayden Ball]

To add, this is easily reproducible following https://academy.nordicsemi.com/simultaneous-updates-for-both-cores-of-the-nrf5340/, then setting `SB_CONFIG_SECURE_BOOT_APPCORE=y`

The resulting diff is the following:

diff --git a/applications/ipc_radio/VERSION b/applications/ipc_radio/VERSION
new file mode 100644
index 000000000..5d04d4847
--- /dev/null
+++ b/applications/ipc_radio/VERSION
@@ -0,0 +1,5 @@
+VERSION_MAJOR = 5
+VERSION_MINOR = 0
+PATCHLEVEL = 2
+VERSION_TWEAK = 0
+EXTRAVERSION =
diff --git a/samples/bluetooth/peripheral_lbs/VERSION b/samples/bluetooth/peripheral_lbs/VERSION
index 5e193c5dc..5d04d4847 100644
--- a/samples/bluetooth/peripheral_lbs/VERSION
+++ b/samples/bluetooth/peripheral_lbs/VERSION
@@ -1,4 +1,4 @@
-VERSION_MAJOR = 3
+VERSION_MAJOR = 5
 VERSION_MINOR = 0
 PATCHLEVEL = 2
 VERSION_TWEAK = 0
diff --git a/samples/bluetooth/peripheral_lbs/app.overlay b/samples/bluetooth/peripheral_lbs/app.overlay
new file mode 100644
index 000000000..fcc2e240f
--- /dev/null
+++ b/samples/bluetooth/peripheral_lbs/app.overlay
@@ -0,0 +1,5 @@
+/ {
+       chosen {
+               nordic,pm-ext-flash = &mx25r64;
+       };
+};
diff --git a/samples/bluetooth/peripheral_lbs/prj.conf b/samples/bluetooth/peripheral_lbs/prj.conf
index 63a681872..a5e69d03e 100644
--- a/samples/bluetooth/peripheral_lbs/prj.conf
+++ b/samples/bluetooth/peripheral_lbs/prj.conf
@@ -15,3 +15,5 @@ CONFIG_BT_LBS_POLL_BUTTON=y
 CONFIG_DK_LIBRARY=y

 CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048
+
+CONFIG_NCS_SAMPLE_MCUMGR_BT_OTA_DFU=y
diff --git a/samples/bluetooth/peripheral_lbs/sysbuild.conf b/samples/bluetooth/peripheral_lbs/sysbuild.conf
new file mode 100644
index 000000000..88e9d7a29
--- /dev/null
+++ b/samples/bluetooth/peripheral_lbs/sysbuild.conf
@@ -0,0 +1,11 @@
+SB_CONFIG_BOOTLOADER_MCUBOOT=y
+SB_CONFIG_PM_EXTERNAL_FLASH_MCUBOOT_SECONDARY=y
+
+SB_CONFIG_SECURE_BOOT_NETCORE=y
+SB_CONFIG_MCUBOOT_UPDATEABLE_IMAGES=2
+SB_CONFIG_NETCORE_APP_UPDATE=y
+SB_CONFIG_MCUBOOT_NRF53_MULTI_IMAGE_UPDATE=y
+SB_CONFIG_MCUBOOT_MODE_OVERWRITE_ONLY=y
+
+SB_CONFIG_SECURE_BOOT_APPCORE=y
+SB_CONFIG_SECURE_BOOT_NETCORE=y
diff --git a/samples/bluetooth/peripheral_lbs/sysbuild/ipc_radio.conf b/samples/bluetooth/peripheral_lbs/sysbuild/ipc_radio.conf
new file mode 100644
index 000000000..7fd7b3d06
--- /dev/null
+++ b/samples/bluetooth/peripheral_lbs/sysbuild/ipc_radio.conf
@@ -0,0 +1,3 @@
+CONFIG_SERIAL=y
+CONFIG_UART_CONSOLE=y
+CONFIG_LOG=y
diff --git a/samples/bluetooth/peripheral_lbs/sysbuild/mcuboot.conf b/samples/bluetooth/peripheral_lbs/sysbuild/mcuboot.conf
new file mode 100644
index 000000000..a567d5488
--- /dev/null
+++ b/samples/bluetooth/peripheral_lbs/sysbuild/mcuboot.conf
@@ -0,0 +1,2 @@
+CONFIG_NORDIC_QSPI_NOR=y
+CONFIG_BOOT_MAX_IMG_SECTORS=256
diff --git a/samples/bluetooth/peripheral_lbs/sysbuild/mcuboot.overlay b/samples/bluetooth/peripheral_lbs/sysbuild/mcuboot.overlay
new file mode 100644
index 000000000..fcc2e240f
--- /dev/null
+++ b/samples/bluetooth/peripheral_lbs/sysbuild/mcuboot.overlay
@@ -0,0 +1,5 @@
+/ {
+       chosen {
+               nordic,pm-ext-flash = &mx25r64;
+       };
+};

Flashing this to a nRF5340 DK, then changing the version numbers in both version files, rebuilding, and attempting an OTA update, you see that the app core version changes in the banner, but the net core does not.

[Hayden Ball]

I _think_ this is fixed by github.com/.../444