Issue with booting from mcuboot. Internal flash does not boot but after waiting for long time boots randomly. Meanwhile External flash setup gives flash read error on mcuboot

[Haseeb Zaib]

NCS -> 2.9.2

Hi, guys

This will be little bit big of a ticket.

So I recently started working with nrf9160, and I have not much dabble with M33 core as well.

But anyways, so I started my project with nrf9160dk board, I have encryptions running on the system (HKDF,HMAC and CHACHA20-Poly1305), so as usual I have my application divided into secure (So keys gets stored here according to my understanding) and non-secure parts, overall the application is built now. So I decided to work on OTA now. 
As I understand the MCUboot is automatically connected to your project using sysbuild and once proper config is enabled the MCUboot will take image from external flash secondary sector or internal flash whatever is set. 

So my first goal was to simply enable mcuboot and see if my image is booted into primary sector, which obv failed at first because I was setting sectors in dts file, which I later got to know that in latest zephyr update , the partitions are being get from partition manager which you have define in pm_static.yml file. After playing around with pm_static.yml file I also studied more about partitioning that in nrf9160 sectors should be aligned with 0x8000, but I would say I finally got the hang of it and using a normal hello_world example I was able to boot into primary application from mcuboot. 

So here is the first part of the problem My application cant boot to primary image from mcuboot

pm_static.yml

#include <autoconf.h>
EMPTY_0:
  address: 0xF000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x1000
EMPTY_1:
  address: 0xE0000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_secondary
  region: flash_primary
  size: 0x20000
app:
  address: 0x68000
  end_address: 0xD8000
  region: flash_primary
  size: 0x70000
mcuboot:
  address: 0x0
  end_address: 0xF000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xF000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x8000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0xD8000
  orig_span: &id001
  - tfm
  - mcuboot_pad
  - app
  region: flash_primary
  sharers: 0x1
  size: 0xC8000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0xD8000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0xC7E00
  span: *id002
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
tfm:
  address: 0x10200
  end_address: 0x68000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x57E00
tfm_secure:
  address: 0x10000
  end_address: 0x68000
  orig_span: &id008
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x58000
  span: *id008
tfm_nonsecure:
  address: 0x68000
  end_address: 0xD8000
  orig_span: &id007
  - app
  region: flash_primary
  size: 0x70000
  span: *id007
mcuboot_secondary:
  address: 0xD8000
  end_address: 0xE0000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x8000
    align_next: 0x8000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size:  0x8000

prj,conf

CONFIG_I2C=y
CONFIG_SENSOR=y
CONFIG_PM_DEVICE=y
CONFIG_CRC=y
CONFIG_SERIAL=y
CONFIG_UART_ASYNC_API=y
CONFIG_UART_INTERRUPT_DRIVEN=y
CONFIG_HW_ID_LIBRARY=y

# Newlib
CONFIG_NEWLIB_LIBC=y

CONFIG_HEAP_MEM_POOL_SIZE=4096
CONFIG_MAIN_STACK_SIZE=4096

CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y

# SPI Flash
CONFIG_SPI=y
CONFIG_SPI_NOR=y
CONFIG_NORDIC_QSPI_NOR=n
# Enable flash operations.
CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_FLASH_PAGE_LAYOUT=y


CONFIG_REBOOT=y 

# Image manager
CONFIG_IMG_MANAGER=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y

mcuboot.conf

# Example of sample specific Kconfig changes when building sample with MCUboot
# when using sysbuild.
CONFIG_MCUBOOT_LOG_LEVEL_WRN=y
CONFIG_MCUBOOT_DOWNGRADE_PREVENTION=y
CONFIG_MCUBOOT_LOG_LEVEL_DEFAULT=y
CONFIG_MCUBOOT_LOG_LEVEL_INF=y
CONFIG_BOOT_MAX_IMG_SECTORS=256
# CONFIG_FLASH=y
# CONFIG_MULTITHREADING=y
# CONFIG_SPI=y
# CONFIG_SPI_NOR=y
# CONFIG_SPI_NOR_FLASH_LAYOUT_PAGE_SIZE=4096

# # Enable flash operations

# CONFIG_PM_OVERRIDE_EXTERNAL_DRIVER_CHECK=y
# CONFIG_PM_EXTERNAL_FLASH_MCUBOOT_SECONDARY=y
# CONFIG_NORDIC_QSPI_NOR=n

sysbuild.conf

# Sysbuild configuration file.

# Enable MCUboot per default for this sample.
SB_CONFIG_BOOTLOADER_MCUBOOT=y
#SB_CONFIG_SECURE_BOOT_APPCORE=y

zephyr generation partitions.yml 

EMPTY_0:
  address: 0xf000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x1000
EMPTY_1:
  address: 0xe0000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_secondary
  region: flash_primary
  size: 0x20000
EMPTY_2:
  address: 0xc2000
  end_address: 0xc8000
  placement:
    after:
    - tfm_its
  region: flash_primary
  size: 0x6000
app:
  address: 0x68000
  end_address: 0xc0000
  region: flash_primary
  size: 0x58000
external_flash:
  address: 0x0
  end_address: 0x800000
  region: external_flash
  size: 0x800000
mcuboot:
  address: 0x0
  end_address: 0xf000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xf000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x8000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0xd8000
  orig_span: &id001
  - tfm
  - mcuboot_pad
  - app
  region: flash_primary
  sharers: 0x1
  size: 0xc8000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0xd8000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0xc7e00
  span: *id002
mcuboot_secondary:
  address: 0xd8000
  end_address: 0xe0000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x8000
    align_next: 0x8000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size: 0x8000
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20016000
  orig_span: &id003
  - tfm_sram
  region: sram_primary
  size: 0x16000
  span: *id003
nrf_modem_lib_ctrl:
  address: 0x20016000
  end_address: 0x200164e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - tfm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_rx:
  address: 0x20018568
  end_address: 0x2001a568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
nrf_modem_lib_sram:
  address: 0x20016000
  end_address: 0x2001a568
  orig_span: &id004
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x4568
  span: *id004
nrf_modem_lib_tx:
  address: 0x200164e8
  end_address: 0x20018568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2080
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
sram_nonsecure:
  address: 0x20016000
  end_address: 0x20040000
  orig_span: &id005
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x2a000
  span: *id005
sram_primary:
  address: 0x2001a568
  end_address: 0x20040000
  region: sram_primary
  size: 0x25a98
sram_secure:
  address: 0x20000000
  end_address: 0x20016000
  orig_span: &id006
  - tfm_sram
  region: sram_primary
  size: 0x16000
  span: *id006
tfm:
  address: 0x10200
  end_address: 0x68000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x57e00
tfm_its:
  address: 0xc0000
  end_address: 0xc2000
  inside:
  - tfm_storage
  placement:
    align:
      start: 0x8000
    before:
    - tfm_otp_nv_counters
  region: flash_primary
  size: 0x2000
tfm_nonsecure:
  address: 0x68000
  end_address: 0xd8000
  orig_span: &id007
  - app
  region: flash_primary
  size: 0x70000
  span: *id007
tfm_otp_nv_counters:
  address: 0xc8000
  end_address: 0xca000
  inside:
  - tfm_storage
  placement:
    align:
      start: 0x8000
    before:
    - tfm_ps
  region: flash_primary
  size: 0x2000
tfm_ps:
  address: 0xd0000
  end_address: 0xd4000
  inside:
  - tfm_storage
  placement:
    align:
      start: 0x8000
    before:
    - end
  region: flash_primary
  size: 0x4000
tfm_secure:
  address: 0x10000
  end_address: 0x68000
  orig_span: &id008
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x58000
  span: *id008
tfm_sram:
  address: 0x20000000
  end_address: 0x20016000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x16000
tfm_storage:
  address: 0xc0000
  end_address: 0xd4000
  orig_span: &id009
  - tfm_ps
  - tfm_its
  - tfm_otp_nv_counters
  region: flash_primary
  size: 0x14000
  span: *id009

The output 

And the funny thing is sometimes it boots into application but after hours of waiting here. As I left it like this yesterday and the application was booted and I again restarted the mcu the application got stuck here


Here is my project folder setup

and here are my project based .conf files which have encryption and other stuff activated
C_PLUS_PLUS.conf

CONFIG_CPP=y
CONFIG_REQUIRES_FULL_LIBCPP=y
CONFIG_STD_CPP20=y
CONFIG_NEWLIB_LIBC=y

crypto.conf

# ---------- TF-M integration ----------
CONFIG_TFM_PROFILE_TYPE_NOT_SET=y     # pick secure services by hand

# ---------- nRF Security + PSA layer ----------
CONFIG_NRF_SECURITY=y
CONFIG_MBEDTLS_PSA_CRYPTO_C=y

# ---------- use CryptoCell hardware driver ----------
CONFIG_PSA_CRYPTO_DRIVER_CC3XX=y
CONFIG_PSA_CRYPTO_DRIVER_OBERON=y

# ---------- algorithms & key-types  ----------
CONFIG_PSA_WANT_ALG_ECDH=y
CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_GENERATE=y
CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_IMPORT=y
CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_EXPORT=y
#CONFIG_PSA_WANT_ECC_CURVE_CURVE25519=y
#following gives Curve25519 support
CONFIG_PSA_WANT_ECC_MONTGOMERY_255=y

#---------- key-derivation & hash for post-ECDH ----------
CONFIG_PSA_WANT_ALG_HKDF=y
CONFIG_PSA_WANT_ALG_SHA_256=y
CONFIG_PSA_WANT_ALG_HMAC=y
CONFIG_PSA_WANT_KEY_TYPE_HMAC=y
#----------- CHACHA20 poly1305 -------------
CONFIG_PSA_WANT_KEY_TYPE_CHACHA20=y
CONFIG_PSA_WANT_ALG_CHACHA20_POLY1305=y

# ---------- entropy source for key generation ----------
CONFIG_PSA_WANT_GENERATE_RANDOM=y

logging.conf

CONFIG_LOG=y
CONFIG_LOG_BUFFER_SIZE=4096
CONFIG_LOG_BACKEND_UART=y   
CONFIG_SHELL_LOG_BACKEND=n    # ← disable log-to-shell



# --- Shell core -------------------------------------------------
CONFIG_SHELL=y
CONFIG_SHELL_STACK_SIZE=5120
CONFIG_SHELL_PROMPT_UART="gasSensor: "
CONFIG_SHELL_CMD_ROOT="gasSensor"
CONFIG_SHELL_BACKEND_SERIAL=y
#CONFIG_SHELL_PROMPT="gas> "          # prompt string

# --- turn OFF all built-in command groups --------------------------
CONFIG_SHELL_CMDS=n           # generic cmds: clear, history, resize, retval
CONFIG_DEVICE_SHELL=n         # device, devmem
CONFIG_DEVMEM_SHELL=n
CONFIG_KERNEL_SHELL=n         # kernel timing, stacks, uptime, ...

# (optional) keep command history?
CONFIG_SHELL_HISTORY=n        # turn off if you don’t need ↑ key recall

lte.conf

CONFIG_NRF_MODEM_LIB=y
CONFIG_LTE_LINK_CONTROL=y
CONFIG_LTE_NETWORK_MODE_LTE_M_NBIOT=y

#ONFIG_AT_HOST_LIBRARY=y


#power saving
CONFIG_LTE_LC_EDRX_MODULE=y
CONFIG_LTE_LC_PSM_MODULE=y
CONFIG_LTE_PSM_REQ=y
CONFIG_LTE_EDRX_REQ=y

mqtt,networking and tls conf

# MQTT
CONFIG_MQTT_LIB=y
CONFIG_MQTT_CLEAN_SESSION=y

# Networking
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=n
CONFIG_NET_SOCKETS_OFFLOAD=y
CONFIG_NET_SOCKETS=y
CONFIG_POSIX_API=y

CONFIG_MODEM_KEY_MGMT=y

CONFIG_MQTT_LIB_TLS=y

I will get back to external flash mcuboot issue later but first I want to boot into my normal application as a normal bootloader

[AHaug]

Hi,

The size of the secondary mcuboot partition has me a bit interested.

The size is 0x8000 in both your defined pm_static and the generated partitioning map, whereas it should be the same size as your primary mcuboot application slot.

I think that maybe this course on our academy pages with theory for DFU in NCS could be helpful https://academy.nordicsemi.com/courses/nrf-connect-sdk-intermediate/lessons/lesson-9-bootloaders-and-dfu-fota/ 

There's lessons (with solutions) for your use case, i.e OTA with external flash and a lesson for LTE OTA (if that's the end goal) as well as BLE OTA.

Let me know if you've reviewed this topic on the pages already or not

Kind regards,
Andreas

[Haseeb Zaib]

yes I gave that size because right now I dont want to load any application to secondary slot.
I only want my system to load into my application thats all, basically I just want right now the mcuboot to jump to my main application and I think it should as it wont get any header on secondary slot and it should jump to primary.

Thats the reason I kept secondary slot that low memory footprint.

As you can see the mcuboot says it jumps and I also mentioned that I left it like this yesterday when I came back to it in the morning surprisingly my application was working

[AHaug]

Ah, I see. Then the reason for the long boot time should not be related to any mismatch in addresses. Anyways, when you get to setting up the external flash and OTA solution, feel free to review those links.

The warning you see in the boot procedure picture you attached stems from this reasoning, at least; MCUboot requires that both the primary and secondary image slots have compatible sector layouts and sizes. If the slots have different sector sizes or counts, MCUboot cannot perform the swap upgrade and will issue this warning. This is especially important when using external flash, as the sector size or count may differ from internal flash.

From what I can see, you've disabled the 3 configurations w.r.t to the secondary slot on the external flash, and no partitions resides on the external flash component you've defined in the pm_static, but there may be something here triggering it nonetheless.

https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/app_dev/bootloaders_dfu/mcuboot_nsib/bootloader_partitioning.html#troubleshooting  showcases the same output window that you attached as well with some hints as to what may be going wrong here

And the funny thing is sometimes it boots into application but after hours of waiting here. As I left it like this yesterday and the application was booted and I again restarted the mcu the application got stuck here

This behavior isn't typical and not something I'm aware others have experience in other cases so the best approach to isolate this issue is to either see if it's resolved if you properly set up the sectors and partition sizes or to do some debugging and find out where in the bootloader the application either gets stuck, hangs or resets

In case you need some resources to get started: https://academy.nordicsemi.com/courses/nrf-connect-sdk-intermediate/lessons/lesson-2-debugging/topic/debugging-in-vs-code/

Kind regards,
Andreas

[Haseeb Zaib]

Thank you for your detail answer.

I understand it, tomorrow I will test it more thoroughly and get back to you.

Although I tested external flash as well and I was getting an issue that mcuboot was not able to read and write to the flash I isolated that issue but will get back to it later after I solve this issue of simply booting into my application

[AHaug]

Great, keep me posted.

To be honest I expect it to be unrelated to the external flash, but since the docs mention the same output that you showcased I think it's worth to isolate if this is the reason or not before we start speculating out of thin air!

Kind regards,
Andreas

[Haseeb Zaib]

okay so after going through some resources and also following nrf connect intermediate guide I was able to use external flash as an secondary storage


This auto generated parition.yml file


As you can see now in output, all the warning are not even showing and mcuboot is showing correct outputs now but still when its jump to first image slot it get stuck and my application is never booted


After testing this I implemented my own static_pm.yml file

#include <autoconf.h>
EMPTY_0:
  address: 0xF000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x1000
app:
  address: 0x68000
  end_address: 0xD8000
  region: flash_primary
  size: 0x70000
mcuboot:
  address: 0x0
  end_address: 0xF000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xF000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x8000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0xD8000
  orig_span: &id001
  - tfm
  - mcuboot_pad
  - app
  region: flash_primary
  sharers: 0x1
  size: 0xC8000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0xD8000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0xC7E00
  span: *id002
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
tfm:
  address: 0x10200
  end_address: 0x68000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x57E00
tfm_secure:
  address: 0x10000
  end_address: 0x68000
  orig_span: &id008
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x58000
  span: *id008
tfm_nonsecure:
  address: 0x68000
  end_address: 0xD8000
  orig_span: &id007
  - app
  region: flash_primary
  size: 0x70000
  span: *id007
external_flash:
  address: 0x0
  device: mx25r64
  end_address: 0x800000
  region: external_flash
  size: 0x800000
mcuboot_secondary:
  address: 0x0
  device: mx25r64
  region: external_flash
  size: 0xC8000

The same response with this one 

So there is definitely something going on with my application code I feel like that does not let mcuboot jump to it

what are your speculation ?

[Haseeb Zaib]

my hunch is that, when the issue is with following crypto.conf file in my project

# ---------- TF-M integration ----------
CONFIG_TFM_PROFILE_TYPE_NOT_SET=y     # pick secure services by hand

# ---------- nRF Security + PSA layer ----------
CONFIG_NRF_SECURITY=y
CONFIG_MBEDTLS_PSA_CRYPTO_C=y

# ---------- use CryptoCell hardware driver ----------
CONFIG_PSA_CRYPTO_DRIVER_CC3XX=y
CONFIG_PSA_CRYPTO_DRIVER_OBERON=y

# ---------- algorithms & key-types  ----------
CONFIG_PSA_WANT_ALG_ECDH=y
CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_GENERATE=y
CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_IMPORT=y
CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_EXPORT=y
#CONFIG_PSA_WANT_ECC_CURVE_CURVE25519=y
#following gives Curve25519 support
CONFIG_PSA_WANT_ECC_MONTGOMERY_255=y

#---------- key-derivation & hash for post-ECDH ----------
CONFIG_PSA_WANT_ALG_HKDF=y
CONFIG_PSA_WANT_ALG_SHA_256=y
CONFIG_PSA_WANT_ALG_HMAC=y
CONFIG_PSA_WANT_KEY_TYPE_HMAC=y
#----------- CHACHA20 poly1305 -------------
CONFIG_PSA_WANT_KEY_TYPE_CHACHA20=y
CONFIG_PSA_WANT_ALG_CHACHA20_POLY1305=y

# ---------- entropy source for key generation ----------
CONFIG_PSA_WANT_GENERATE_RANDOM=y

maybe CONFIG_TFM_PROFILE_TYPE_NOT_SET=y 

it is not sitting well with mcuboot ?

[Haseeb Zaib]

Also I tried external flash with hello_world program and my system immediately booted into application


Also I added my above crypto.conf to project and hello_world application still booted



Also I kept same partitions for both projects hello_world and my main application



I am using external flash in my application as well to store certificates may that could be an issue ?? I kept address space different ofcourse but still



Inside this library I am using normal flash_write,erase and read functions


BTW here is my code https://github.com/haseebzaib/GasSensor_Device
maybe will be easier to test on your side as well