After upgrading mcuboot, the device crashes upon power-up

RE: After upgrading mcuboot, the device crashes upon power-up

Hieu — Tue, 25 Mar 2025 21:27:46 GMT

This is a follow-up for future readers of this case.

The issue is that there is a feature in NSIB and MCUboot where MCUboot can use the crypto library in NSIB.

This feature is enabled by default when using Sysbuild (Ref this and this).
In NCS v2.6.1, Sysbuild isn't the default build method, and it isn't enabled. B0 and MCUboot are built without supporting this feature.
In NCS v2.9.0, Sysbuild is the default method, and MCUboot is built in a way that needs this feature.

When the v2.9.0 MCUboot is flashed onto a device with default v2.6.1 NSIB, it cannot find the crypto library it needs and therefore fails to start.
During the first reset where MCUboot still works that you observe, the MCUboot running is actually the old v2.6.1 one.
During this reset, it swaps the MCUboot from secondary slot into position. The new v2.9.0 MCUboot only runs (and fails) from the second reset.

Currently, the MCUboot-use-b0-crypto feature is always enabled by default in a NSIB+MCUboot+Sysbuild setup.

For projects that need to upgrade MCUboot to a system without B0 external crypto support, developers can comment out this CMake section as a workaround:
https://github.com/nrfconnect/sdk-nrf/blob/v2.9.0/sysbuild/CMakeLists.txt#L467-L472.

Another workaround is to build with the old parent-child multi-image system instead of Sysbuild.

The issue has been reported, and a fix will be done.

RE: After upgrading mcuboot, the device crashes upon power-up

Harvey Zheng — Mon, 17 Mar 2025 08:30:09 GMT

Hi,

I want to follow up in the new ticket.

Thank you.

RE: After upgrading mcuboot, the device crashes upon power-up

Hieu — Mon, 17 Mar 2025 08:27:44 GMT

Hi,

Do you want to follow up in the new ticket or here? I am trying to confirm a few things before replying to you.

RE: After upgrading mcuboot, the device crashes upon power-up

Harvey Zheng — Mon, 17 Mar 2025 07:33:28 GMT

Hi,

For some reason, I resubmitted a private ticket. It can be seen here

Thanks

RE: After upgrading mcuboot, the device crashes upon power-up

Harvey Zheng — Thu, 13 Mar 2025 08:11:53 GMT

A bit of additional information:
I compared the .config files generated by NCS2.6.1 MCUBoot and NCS2.9.0 MCUBoot. They seem to use different crypto, this is because bootloader\mcuboot\boot\zephyr\external_crypto.conf was merged when compiling NCS2.9.0, which wasn't there when NCS2.6.1 was build!

Will this affect B0's validation of MCUBoot?

RE: After upgrading mcuboot, the device crashes upon power-up

Harvey Zheng — Thu, 13 Mar 2025 07:57:44 GMT

Hi,

[quote userid="9456" url="~/f/nordic-q-a/119311/after-upgrading-mcuboot-the-device-crashes-upon-power-up/527082"]I forgot to mention. These should be configured in the MCUboot image only.[/quote]

Since enabling DEBUG Logging in NCS2.6.1 MCUBoot would take up a lot of space, after trying it out I have not enabled it in the NCS2.6.1 example for now, and I will try to change the partition size if necessary.

I had an important discovery in today's test, and synchronize it here

I set the following KCONFIG in the NCS2.6.1 B0 configuration to y and it then successfully upgraded NCS2.6.1 MCUBoot to NCS2.9.0 MCUBoot.

CONFIG_BL_ROT_VERIFY_EXT_API_ENABLED=y
CONFIG_BL_SHA256_EXT_API_ENABLED=y
CONFIG_BL_SECP256R1_EXT_API_ENABLED=y
CONFIG_BL_VALIDATE_FW_EXT_API_ENABLED=y
CONFIG_EXT_API_PROVIDE_EXT_API_ENABLED=y

The screenshot is the successful upgrade logging after I modified NCS2.6.1 B0.

There are two questions about this:
1. will these KCONFIG options have any effect on the upgrade? I'm guessing it will have an effect on mcuboot firmware validation?
2. In the products we have produced, these KCONFIG options are disabled and B0 cannot be upgraded. I've tried setting the above options to n in NCS2.9.0 B0 to ensure consistency with the NCS2.6.1 example, but this causes it to stay at B0 after powering up, and not be able to jump to MCUBoot, which looks like the same phenomenon as the NCS2.6.1 MCUBoot upgrade to NCS2.9.0 failing. So how can I upgrade MCBoot to NCS2.9.0 without updating NCS2.6.1 B0?

RE: After upgrading mcuboot, the device crashes upon power-up

Hieu — Thu, 13 Mar 2025 06:42:45 GMT

[quote user="Magdalena"]

Hieu said:
CONFIG_LOG=y
CONFIG_LOG_MODE_DEFERRED=y
CONFIG_LOG_DEFAULT_LEVEL=4
CONFIG_MULTITHREADING=y
CONFIG_MCUBOOT_LOG_LEVEL_DBG=y

I will try it for more helpful logging, thank you!

[/quote]

I forgot to mention. These should be configured in the MCUboot image only.

RE: After upgrading mcuboot, the device crashes upon power-up

Harvey Zheng — Thu, 13 Mar 2025 03:58:29 GMT

Hi,

[quote userid="9456" url="~/f/nordic-q-a/119311/after-upgrading-mcuboot-the-device-crashes-upon-power-up/527049"]Is the screenshot with logging from a failure case?[/quote]

Yes, all failed log messages are from two attachments tested on nRF9160DK: ncs2.6.1.zip and ncs2.9.0.zip

[quote userid="9456" url="~/f/nordic-q-a/119311/after-upgrading-mcuboot-the-device-crashes-upon-power-up/527049"]CONFIG_LOG=y
CONFIG_LOG_MODE_DEFERRED=y
CONFIG_LOG_DEFAULT_LEVEL=4
CONFIG_MULTITHREADING=y
CONFIG_MCUBOOT_LOG_LEVEL_DBG=y[/quote]

I will try it for more helpful logging, thank you!

RE: After upgrading mcuboot, the device crashes upon power-up

Hieu — Wed, 12 Mar 2025 20:55:44 GMT

Hi Harvey,

Is the screenshot with logging from a failure case?

From the screenshot in your first reply, it looks like you are running without MCUboot logging at all. There seem to be only logs from the application and NSIB. There might be clue in MCUboot logging. Could you please enable log level debug for MCUboot?

You will need these configurations:

CONFIG_LOG=y
CONFIG_LOG_MODE_DEFERRED=y
CONFIG_LOG_DEFAULT_LEVEL=4
CONFIG_MULTITHREADING=y
CONFIG_MCUBOOT_LOG_LEVEL_DBG=y

Another thing I want to consider is looking at hex dumps. It might be of interest to dump the entire memory at these points and track the changes:

Before new MCUboot download
After new MCUboot download, before first reset
Right after first reset
Right after second reset

Hieu

RE: After upgrading mcuboot, the device crashes upon power-up

Harvey Zheng — Mon, 10 Mar 2025 03:59:12 GMT

Hi,

[quote userid="106736" url="~/f/nordic-q-a/119311/after-upgrading-mcuboot-the-device-crashes-upon-power-up/526318"]Looking at the examples you sent, pm_static.yml is mostly commented out.[/quote]

Yes, I only defined the location of the mcuboot secondary partition, the others are generated at build time, and I've compared the partitions of the two examples using memory report, and they are exactly the same.

[quote userid="106736" url="~/f/nordic-q-a/119311/after-upgrading-mcuboot-the-device-crashes-upon-power-up/526318"]A pm_static.yml that defined all DFU partitions is required. Try to use the same pm_static.yml for both projects. Does DFU work then?[/quote]

I tried to use the exact same pm_static.yml in both examples, which defines all partitions, but still the upgrade fails with the same occurrences.

On this basis, I tried to keep the B0 configuration of the NCS 2.9.0 example consistent with the B0 configuration of the NCS 2.6.1 example, mainly by setting the following KCONFIG. Then flash the NCS2.9.0 example directly, and the result is the same, it seems to be unable to start MCUBoot after running B0.

CONFIG_BL_ROT_VERIFY_EXT_API_ENABLED=n
CONFIG_BL_SHA256_EXT_API_ENABLED=n
CONFIG_BL_SECP256R1_EXT_API_ENABLED=n
CONFIG_BL_VALIDATE_FW_EXT_API_ENABLED=n
CONFIG_EXT_API_PROVIDE_EXT_API_ENABLED=n

RE: After upgrading mcuboot, the device crashes upon power-up

Sigurd Hellesvik — Fri, 07 Mar 2025 10:07:51 GMT

Looking at the examples you sent, pm_static.yml is mostly commented out.

A pm_static.yml that defined all DFU partitions is required. Try to use the same pm_static.yml for both projects. Does DFU work then?

RE: After upgrading mcuboot, the device crashes upon power-up

Harvey Zheng — Fri, 07 Mar 2025 07:29:06 GMT

Hi Sigurd Hellesvik,

Is there any progress on this issue so far and what further information is needed from my side?

Thanks

RE: After upgrading mcuboot, the device crashes upon power-up

Harvey Zheng — Wed, 05 Mar 2025 03:45:03 GMT

Hi,

I took over the problem and trying to fix him. I've reorganized the problem points.

Problem Description

Our application uses MCUBoot as an upgradeable bootloader in the order NSIB(B0) -> MCUBoot -> Application, now we adapted NCS2.9.0 and tried to upgrade MCUBoot from NCS2.6.1 to NCS2.9.0, after the device receives the upgrade package, confirms it and reset, the log shows that MCUBoot has carried the new upgrade package to the S1 partition, and the S1 partition version read on the application is 2. However, when I reset the device again, the program fails to run to the application, and it seems that the problem has already occurred at B0 or MCUBoot.

Examples

I wrote two examples using NCS2.6.1 and NCS2.9.0 and reproduced it on the nRF9160DK.

Differences between the two test examples:

- NCS2.6.1 examples : Sysbuild is not used.
- NCS2.9.0 examples : Sysbuild is used.

Differences between NCS2.6.1 examples and my application:

- NCS2.6.1 Sample: B0 and MCUBoot configurations are basically the same as my application, the first difference is that the MCUBoot partition size is 0xBE00 to be consistent with NCS2.9.0 example(so far testing has shown that MCUBoot sizes of either 0xBE00 or 0xC000 both cause the above problems); The second difference is that the upgrade package is transmitted using the MCUMGR UART.

Recurrence process and log

1. Compile NCS2.6.1 examples with NCS2.6.1 SDK, program into nRF9160DK.

2. Compile NCS2.9.0 examples with NCS2.9.0 SDK.

3. The uart1 port is used by MCUMGR, running a command line in the NCS2.9.0 example directory to transfer the S1 upgrade package to the nRF9160DK

mcumgr -t 60 --conntype serial --connstring=COM48 image upload .\build\signed_by_mcuboot_and_b0_s1_image.bin

4. Get firmware information

mcumgr -t 60 --conntype serial --connstring=COM48 image list

5. Confirm the firmware

mcumgr -t 60 --conntype serial --connstring=COM48 image confirm <hash> true

6. Press the reset button to reset the DK, after booting you can see that the S1 version has been updated to 2.
7. Press the reset button again, the device does not run to the application, and the logs show that it no longer logs after B0 is ready to jump to the S1 partition (the new MCUBoot firmware for S1 already has log printing enabled).

The two attachments are examples of my tests:

devzone.nordicsemi.com/.../ncs2.6.1.zip

devzone.nordicsemi.com/.../ncs2.9.0.zip

RE: After upgrading mcuboot, the device crashes upon power-up

Sigurd Hellesvik — Wed, 26 Feb 2025 14:56:46 GMT

devzone.nordicsemi.com/.../0447.hello_5F00_world.zip

RE: After upgrading mcuboot, the device crashes upon power-up

Leo — Wed, 26 Feb 2025 14:45:57 GMT

Can you provide the example you added?

RE: After upgrading mcuboot, the device crashes upon power-up

Sigurd Hellesvik — Wed, 26 Feb 2025 14:36:01 GMT

When I add NSIB and MCUboot to hello world, it looks like this:

A: NSIB logs

B: MCUboot logs

C: Application logs

So for the DK, it logs by default at least

RE: After upgrading mcuboot, the device crashes upon power-up

Leo — Wed, 26 Feb 2025 11:57:17 GMT

How to view the logs of NSIB?

RE: After upgrading mcuboot, the device crashes upon power-up

Sigurd Hellesvik — Wed, 26 Feb 2025 11:53:25 GMT

Do you have logs from NSIB?

RE: After upgrading mcuboot, the device crashes upon power-up

Leo — Wed, 26 Feb 2025 11:44:45 GMT

I tried copying build/partitions.yml into pm_static.yml.After upgrading mcuboot from NCS 2.6.1 to NCS 2.9.0, the upgrade file is downloaded, and the device restarts and runs normally. The mcuboot version number and active slot are updated as expected. However, when the device is powered off and then powered on again, it fails to work properly, almost as if it has crashed.

RE: After upgrading mcuboot, the device crashes upon power-up

Sigurd Hellesvik — Wed, 26 Feb 2025 11:39:54 GMT

Hi,

[quote user=""]In NCS 2.6.1, the default value of CONFIG_PM_PARTITION_SIZE_MCUBOOT is 0xc000, while in NCS 2.9.0, the default value is 0xbe00. A bug related to the MCUboot partition size was fixed on October 24, 2024. Could this bug have affected the upgrade failure?[/quote]

Generally, you need to follow the Static partitioning requirement. Im not 100% sure since it is just the headers in this case.

But try to copy build/partitions.yml into pm_static.yml and then build the project for v2.9.0. Does DFU work then?

Regards,
Sigurd Hellesvik