Upgrade SDK 2.1.2 to SDK 2.2.0+ leads to boot loop (in FCB settings storage)

[CRD]

We are trying to upgrade our SDK from 2.1.2 to 2.2.0 then 2.30 but we have encountered a very hard to track down bug.

I cannot share the actual app code but the core is very similar to the nrf asset tracker v2 (uses mcuboot). 

What's wrong:

- The app builds and runs great on SDK 2.1.2

- The app builds but does not run on SDK 2.2.0 or SDK. 2.3.0. We get an endless loop (mcuboot runs fine and can be interrupted for serial recovery) but the app crashes silently before it can print or debug anything.

What we tried:

I followed all the update notes from this thread: devzone.nordicsemi.com/.../problems-migrating-nrf5340-project-from-sdk-1-9-1-to-2-3-0

We still have a silent boot loop....

After a TON of digging and debugging I tracked it down to a section of code. We use FCB settings storage and if we comment out the line 

`err = settings_subsys_init();` we no longer get the immediate bootloop but then of course our settings don't work.

 

I dug even deeper and discovered that if we re-enable `err = settings_subsys_init();` but add to our overlay the definition:

/ {
    chosen {
        zephyr,settings-partition=&storage_partition;
    };
};

We can now boot the app but we now get a crash for <err> flash_nrf: unaligned address: 0x00014200:15872

We are getting somewhere...

 

After a lot more digging I tracked it down to the following but I can't figure out anything futher.

zephyr > soc > arm > nordic_nrf > common > soc_secure.c

in the function at line 30 (int soc_secure_mem_read(void *dst, void *src, size_t len))

the call to tfm_platform_mem_read(dst, (uintptr_t)src, len, &result); returns 

TFM_PLATFORM_ERR_INVALID_PARAM

This is throwing off the whole FCB initialization sequence which leads to the memory address issue. I check in sdk 2.1.2 and we do not encounter this error despite the code and libraries seemingly not changing much.

Are we missing some new config? This is blocking any upgrade to v2.2.0 or beyond. I have included some relevant configs below...

proj.conf includes:

# Settings - Used to store real-time device configuration to flash.
CONFIG_FCB=y
CONFIG_SETTINGS=y
CONFIG_SETTINGS_FCB=y

# Flash - Used in FOTA, settings and storage
CONFIG_FLASH=y
CONFIG_FLASH_PAGE_LAYOUT=y
CONFIG_FLASH_MAP=y
CONFIG_STREAM_FLASH=y
CONFIG_MPU_ALLOW_FLASH_WRITE=y

# MCUBOOT
CONFIG_BOOTLOADER_MCUBOOT=y
CONFIG_IMG_MANAGER=y
CONFIG_MCUBOOT_IMG_MANAGER=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y
CONFIG_SECURE_BOOT=y
CONFIG_BUILD_S1_VARIANT=y

 

&flash0 {

    partitions {
        compatible = "fixed-partitions";
        #address-cells = <1>;
        #size-cells = <1>;

        boot_partition: partition@0 {
            label = "mcuboot";
            reg = <0x00000000 0x10000>;
        };
        slot0_partition: partition@10000 {
            label = "image-0";
        };
        slot0_ns_partition: partition@50000 {
            label = "image-0-nonsecure";
        };
        slot1_partition: partition@80000 {
            label = "image-1";
        };
        slot1_ns_partition: partition@c0000 {
            label = "image-1-nonsecure";
        };
        scratch_partition: partition@f0000 {
            label = "image-scratch";
            reg = <0x000f0000 0xa000>;
        };
        storage_partition: partition@fa000 {
            label = "storage";
            reg = <0x000fa000 0x00006000>;
        };
    };
};

 

I'm guessing that there must be some kind of update to TFM or to the FCB flash region that we are no longer compatible with. Any help would be greatly appreciated. Thank you.

[CRD]

I'm pretty sure I found the solution to our issue. It's probably unique to our project/situation but I'll share in case it helps someone else.

It basically looks like we accidentally added some extra configs into our project at some point that are now conflicting on version 2.2.0

To fix the bootloop we first removed this change:

/ {
   chosen {
     zephyr,settings-partition=&storage_partition;
   };
};

Then we removed these from our config. Apparently this is fine in SDK 2.1.2 but not in 2.2.0 or higher if you are also using FCB. 

# NVS 
CONFIG_PARTITION_MANAGER_ENABLED=y
CONFIG_FLASH_MAP_CUSTOM=y
CONFIG_NVS=y

Now the build works and the app boots normally. 

[Susheel Nuguru]

CRD said:

Then we removed these from our config. Apparently this is fine in SDK 2.1.2 but not in 2.2.0 or higher if you are also using FCB. 

I am not sure what is conflicting here, but it is best to get an indepth overview of why this change is working for you. Just to be sure that no other unwanted dependencies are pulled in.

[CRD]

I think it may have just been this line that was causing the issue:

CONFIG_PARTITION_MANAGER_ENABLED=y