MCUBoot: Image in the primary slot is not valid!

[Farhang]

Hi Nordic Team,

I have a custom application running based on nRF NCS 1.5.0 with MCUboot enabled.

I'm having an issue where MCUboot simply doesn't jump to the application.
This comes and goes when I add/remove code from my main application... but with no noticible pattern. I wonder if image size is a factor and thing are overlapping in Flash image...

only using 42% of Flash for app.

1> Memory region Used Size Region Size %age Used
1> FLASH: 207828 B 495104 B 41.98%
1> SRAM: 66368 B 256 KB 25.32%
1> IDT_LIST: 88 B 2 KB 4.30%

Running partition manager report on build...\partitions.yml

I believe this is the standard setup where MCUboot take up to 48K and 2 app banks

flash_primary (0x100000 - 1024kB):
+-------------------------------------------------+
| 0x0: mcuboot (0xc000 - 48kB) |
+---0xc000: mcuboot_primary (0x79000 - 484kB)-----+
| 0xc000: mcuboot_pad (0x200 - 512B) |
+---0xc200: mcuboot_primary_app (0x78e00 - 483kB)-+
| 0xc200: app (0x78e00 - 483kB) |
+-------------------------------------------------+
| 0x85000: mcuboot_secondary (0x79000 - 484kB) |
| 0xfe000: settings_storage (0x2000 - 8kB) |
+-------------------------------------------------+

sram_primary (0x40000 - 256kB):
+--------------------------------------------+
| 0x20000000: sram_primary (0x40000 - 256kB) |
+--------------------------------------------+

I can get more debug but here's some I got via enabling minimal logging via RTT in MCUboot.conf

*** Booting Zephyr OS build v2.4.99-ncs1 ***
I: Starting bootloader
I: det pin = 12 rc = 1
I: Primary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Secondary image: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
I: Boot source: none
I: Swap type: none
E: Image in the primary slot is not valid!
E: fih_rc=1, succ=0
E: Unable to find bootable image

One clue is.... I started a bit of investigation into "merged.hex" and how it is created. I was surprised to see mergehex.py combines 4 different images with argument "--overlap=replace"

C:\Python38\python.exe C:/depot/ncs/zephyr/scripts/mergehex.py -o C:/depot/application/build_nrf52840dk_nrf52840/zephyr/merged.hex --overlap=replace C:/depot/application/build_nrf52840dk_nrf52840/mcuboot/zephyr/zephyr.hex C:/depot/application/build_nrf52840dk_nrf52840/zephyr/mcuboot_primary.hex C:/depot/application/build_nrf52840dk_nrf52840/zephyr/zephyr.hex C:/depot/application/build_nrf52840dk_nrf52840/zephyr/app_signed.hex

When I looked at those 4 images there is a LOT of overlap. Shouldn't the final image just consist of mcuboot image and app_signed?

For example: "app_signed.hex" and "zephyr.hex" are almost identical, they start the same but app_signed.hex has signature at the end which is normal, see below for similarities at start.

This is strange to me why zephyr.hex needs to be included in the mergehex while app_signed has everything as far as the app goes, or is there a piece that I am missing? 

This is just one clue, but I haven't determined if this is causing the MCUboot not jumping to app. In other times when it does work, the overlap/replace redundancies are still in place. So I'm not sure if this is the cause.

Thanks for your assistance in advance, this is blocking progress of development for us.

[Farhang]

Simon I forgot to mention, I do a clean/ full rebuild, so I assume all hashes are recalculated properly, no?

I will do more debugging as you suggested and report back.

RE: overlap=replace, I didn't change the build to this option, it was already set to that. Is that standard or not?

I guess I can look at an sample/example and see how merge is done for a standard project.

[Simon]

Farhang said:

Simon I forgot to mention, I do a clean/ full rebuild, so I assume all hashes are recalculated properly, no?

I will do more debugging as you suggested and report back.

Okay, sounds good. I'll be waiting for your report

Farhang said:

RE: overlap=replace, I didn't change the build to this option, it was already set to that. Is that standard or not?

I guess I can look at an sample/example and see how merge is done for a standard project.

Yes, it's standard. It will make it such that the app_signed.hex file is used instead of the zephyr.hex file. Check my answer here for a more detailed explanation:

"merged.hex will contain the files <your sample>\<build folder>\mcuboot\zephyr\zephyr.hex as well as <your sample>\<build folder>\zephyr\app_signed.hex. If you're building a nonsecure application (nrf9160dk_nrf9160ns) the merged hex file will include <your sample>\<build folder>\spm\zephyr\zephyr.hex also. When you program the merged.hex file, it will program all the mentioned hex files at once."

Best regards,

Simon

[Farhang]

Simon Thanks for the info above. 

You did understand my issue, but I guess 3 things you added are different hence one of these may have fixed the issue.

- Debug optimizations on - which probably expands the size

- larger FLASH partition for MCUBoot

- Ozone rather than SES 5.34a

I will try above but 2 follow up questions:

1- Is Ozone the preferred/recommended debugger IDE for nRF NCS or is SES?

2- Do you know which of the above fixed the issue?

[Simon]

Farhang said:

1- Is Ozone the preferred/recommended debugger IDE for nRF NCS or is SES?

Ozone is free to use with our development kits, so both SES and Ozone are valid options to debug with. 

Farhang said:

2- Do you know which of the above fixed the issue?

I tried to comment out the options below, and then set 7-8 breakpoints in bootloader/mcuboot/boot/zephyr/main.c, but I did not encounter any issues with the validation.

CONFIG_DEBUG_OPTIMIZATIONS=y
CONFIG_PM_PARTITION_SIZE_MCUBOOT=0x10000

I will do some more testing with SES later on

[paat]

I experienced same or similar issue with MCUBoot checking the primary slot hash code and failing when debugging from SES. The workaround or solution that worked for me was to disable the check in MCUBoot conf:

CONFIG_BOOT_VALIDATE_SLOT0=n

[Farhang]

I also came across this setting yesterday - and suspected could be the fix for this issue.

It lines up with what Simon and I found regarding breakpoint instructions in Thumb:

"It seems like 0xBE00 is the thumb instruction for a break point: https://interrupt.memfault.com/blog/cortex-m-breakpoints. "

Unrelated, but Simon any idea why this setting is not included in the master list of KConfigs?

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/kconfig/index-all.html

There are many other settings with MCUboot that I can't find in there either such as CONFIG_BOOT_SWAP_USING_MOVE

From what I understand that list should include all the kconfigs in the entire NCS - no?

[Simon]

Farhang said:

Unrelated, but Simon any idea why this setting is not included in the master list of KConfigs?

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/kconfig/index-all.html

I think this is because CONFIG_BOOT_VALIDATE_SLOT0 is defined in the MCUboot sample https://github.com/nrfconnect/sdk-mcuboot/blob/4e4e979c1998e6dad7e47011e6e076b861921645/boot/zephyr/Kconfig#L178 

In https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/kconfig/index-all.html, you will only find the Kconfigs defined in Zephyr, nrf and nrfxlib.

Best regards,

Simon