MCUBOOT won't boot image in 2.4.0

[Anthony Ambuehl]

I have an application I have been developing in NCS v2.3.0.  I'm trying now to updata to v2.4.0.

The application targets NRF5340.

It uses NSIB (b0/b0n) and for the application processor also includes mcuboot.

It also uses TF-M and I have implemented a custom service in a secure partition for security features.

It is also configured for mcuboot to use the external flash partition.

The application is based on smp_server and I uses different build configurations to build it either with the console or with smp / mcumgr support.

I have found it necessary to make quite a changes to be compatible with v2.4.0:  Secure partitions have a bunch of changes and I had to change a variety of Kconfig variables as well.  Since my application is drawing from multiple different samples, its hard to know for sure what other changes I may need.

I was eventually able to get the compile to complete cleanly, but I'm finding the mcuboot doesn't find any valid/bootable partition.

In addition no matter what I try, I cannot disable the LOG_MODE_MINIMAL from mcuboot to get more detailed log information.  Even though in child_image/mcuboot.conf has CONFIG_LOG_MODE_MINIMAL=n    the generated mcuboot/zephyr/.config   says CONFIG_LOG_MODE_MINIMAL=y .   I really don't understand why it switches to this value when I specifically requested the opposite.

As well, I cannot debug, since every time I  try to run the debug action in VSCode, I get a write verification failure.

Erasing page at address 0xAB000.
Erasing page at address 0xAC000.
Erasing page at address 0xAD000.
WARNING: A UICR write operation has been requested but UICR has not been
WARNING: erased. Please verify that the result is correct.
Applying system reset.
Checking that the area to write is not protected.
Programming device.
Verifying programming.
ERROR: Write verify failed.
FATAL ERROR: command exited with status 25: nrfjprog --program 'd:\AAA\prov\provisioning_image\smp_svr\build_shell\zephyr\GENERATED_CP_APPLICATION_merged_domains.hex' --sectorerase --verify -f NRF53 --coprocessor CP_APPLICATION --snr 1050031166

* The terminal process terminated with exit code: 25.
* Terminal will be reused by tasks, press any key to close it.

So in summary no logs, no debug, and it won't boot.

How should I proceed with resolving this problem?

[Anthony Ambuehl]

I modified my config, removing:

CONFIG_LOG_DEFAULT_LEVEL=0

and I now get a few more relevant logs.  I further extended the logging in bootloader/mcuboot/boot/bootutil/loader.c, adding logs:

    BOOT_HOOK_CALL_FIH(boot_image_check_hook, FIH_BOOT_HOOK_REGULAR,
                       fih_rc, BOOT_CURR_IMG(state), slot);
    BOOT_LOG_ERR("boot_image_check_hook returned %d", fih_rc);
    if (FIH_EQ(fih_rc, FIH_BOOT_HOOK_REGULAR))
    {
        FIH_CALL(boot_image_check, fih_rc, state, hdr, fap, bs);
        BOOT_LOG_ERR("boot_image_check returned %d", fih_rc);
    }
    if (!boot_is_header_valid(hdr, fap) || FIH_NOT_EQ(fih_rc, FIH_SUCCESS))
    {
        if ((slot != BOOT_PRIMARY_SLOT) || ARE_SLOTS_EQUIVALENT())
        {
            flash_area_erase(fap, 0, flash_area_get_size(fap));
            /* Image is invalid, erase it to prevent further unnecessary
             * attempts to validate and boot it.
             */
        }
#if !defined(__BOOTSIM__)
        BOOT_LOG_ERR("Image in the %s slot is not valid!",
                     (slot == BOOT_PRIMARY_SLOT) ? "primary" : "secondary");
        BOOT_LOG_ERR("header %s valid and rc is %d",
                     (boot_is_header_valid(hdr, fap) ? "is" : "is not"),
                     fih_rc);
#endif

and here are the logs I'm getting 

*** Booting Zephyr OS build v3.3.99-ncs1 ***
I: Starting bootloader
I: Swap type: none
I: Swap type: none
E: boot_image_check_hook returned 1
E: Image in the primary slot is not valid!
E: header is valid and rc is 1
E: Unable to find bootable image

The log indicates the boot_image_check_hook is returning 1 (e.g. FIH_BOOT_HOOK_REGULAR)  but for some reason I'm not getting the log reporting the return value of boot_image_check..

[Anthony Ambuehl]

It looks like wherever boot_image_check_hook is coming from, it has not been updated to work with the Fault Injection Hardening FIH.  FIH is changing the FIH_BOOT_HOOK_REGULAR to 0x19C1F6E1 as I can see from this modified log:

*** Booting Zephyr OS build v3.3.99-ncs1 ***
I: Starting bootloader
I: Swap type: none
I: Swap type: none
E: boot_image_check_hook returned 0x1 regular is 0x19c1f6e1
E: Image in the primary slot is not valid!
E: header is valid and rc is 1
E: Unable to find bootable image

Are there different boot_image_check_hooks for nrf5340 ? 

Do I need to disable the FIH profile for mcuboot to work?

[Anthony Ambuehl]

Confirmed...

v2.4.0/nrf/modules/mcuboot/hooks/nr53_hooks.c is returning BOOT_HOOK_REGULAR instead of FIH_BOOT_HOOK_REGULAR.

I'm not clear on how/when nrf53_hooks.c gets pulled in, but it seems it has not been tested with mcuboot with FIH.

[Vidar Berg]

Thanks for the updates. I notice that the 'nrf5340dk_nrf5340_cpuapp_ns' target is not being tested for the smp_srv sample; only the 'nrf5340dk_nrf5340_cpuapp' target is being tested. If you patch nrf53_hooks.c, would you be able to make it work for now? I'll discuss this further with the developers when they return to the office on Monday.

[Anthony Ambuehl]

Yes I can confirm that after modifying nrf53_hooks.c mcuboot detected the primary slot as valid.

It would be great if in addition to fixing this, they actually included the nrf5340dk_nrf5340_cpuapp_ns in the regression testing.  

Also it would be wonderful if somehow VSCode would be aware of nrf53_hooks., I had to a perform a global grep of the entire v2.4.0 code base from the shell to locate nrf53_hooks.c, and its still unclear to me how/when this files is used.  With SES at least we always had easy access to all of the source code that was included.  Now its a complete mystery. 

Even worse VSCode actually shows incorrect macro values, I have to determine the correct values by logging them at run time.  For example the value of FIH_BOOT_HOOK_REGULAR reports as 1 in VSCode but reports as 0x19c1f6e1 at run time.  I suppose this is probably an artifact of child images having different config settings.. there needs to be a way to deal with that.

[Sigurd Hellesvik]

Hi Anthony,

I will continue helping in this case.

Anthony Ambuehl said:

Also it would be wonderful if somehow VSCode would be aware of nrf53_hooks., I had to a perform a global grep of the entire v2.4.0 code base from the shell to locate nrf53_hooks.c, and its still unclear to me how/when this files is used.  With SES at least we always had easy access to all of the source code that was included.  Now its a complete mystery. 

Even worse VSCode actually shows incorrect macro values, I have to determine the correct values by logging them at run time.  For example the value of FIH_BOOT_HOOK_REGULAR reports as 1 in VSCode but reports as 0x19c1f6e1 at run time.  I suppose this is probably an artifact of child images having different config settings.. there needs to be a way to deal with that.

For what it's worth, I agree that our extensions search features have potential for improvement.
I will convey your feedback to our VS Code team, thank you for sharing your thoughts!

Seems to me that you have solved some of the issues in this case.
To avoid misunderstandings:
Do you need help with anything else in this ticket?

Regards,
Sigurd Hellesvik

[Anthony Ambuehl]

Primary issue with mcuboot not working when running with nrf5340dk_nrf5340_cpuapp_ns is resolved after fixing the issue in nrf53_hooks. 

I hope that Nordic can add more testing to the releases to avoid future regressions.