Issue with booting from mcuboot. Internal flash does not boot but after waiting for long time boots randomly. Meanwhile External flash setup gives flash read error on mcuboot

NCS -> 2.9.2

Hi, guys

This will be little bit big of a ticket.

So I recently started working with nrf9160, and I have not much dabble with M33 core as well.

But anyways, so I started my project with nrf9160dk board, I have encryptions running on the system (HKDF,HMAC and CHACHA20-Poly1305), so as usual I have my application divided into secure (So keys gets stored here according to my understanding) and non-secure parts, overall the application is built now. So I decided to work on OTA now. 
As I understand the MCUboot is automatically connected to your project using sysbuild and once proper config is enabled the MCUboot will take image from external flash secondary sector or internal flash whatever is set. 

So my first goal was to simply enable mcuboot and see if my image is booted into primary sector, which obv failed at first because I was setting sectors in dts file, which I later got to know that in latest zephyr update , the partitions are being get from partition manager which you have define in pm_static.yml file. After playing around with pm_static.yml file I also studied more about partitioning that in nrf9160 sectors should be aligned with 0x8000, but I would say I finally got the hang of it and using a normal hello_world example I was able to boot into primary application from mcuboot. 

So here is the first part of the problem My application cant boot to primary image from mcuboot

pm_static.yml

#include <autoconf.h>
EMPTY_0:
  address: 0xF000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x1000
EMPTY_1:
  address: 0xE0000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_secondary
  region: flash_primary
  size: 0x20000
app:
  address: 0x68000
  end_address: 0xD8000
  region: flash_primary
  size: 0x70000
mcuboot:
  address: 0x0
  end_address: 0xF000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xF000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x8000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0xD8000
  orig_span: &id001
  - tfm
  - mcuboot_pad
  - app
  region: flash_primary
  sharers: 0x1
  size: 0xC8000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0xD8000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0xC7E00
  span: *id002
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
tfm:
  address: 0x10200
  end_address: 0x68000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x57E00
tfm_secure:
  address: 0x10000
  end_address: 0x68000
  orig_span: &id008
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x58000
  span: *id008
tfm_nonsecure:
  address: 0x68000
  end_address: 0xD8000
  orig_span: &id007
  - app
  region: flash_primary
  size: 0x70000
  span: *id007
mcuboot_secondary:
  address: 0xD8000
  end_address: 0xE0000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x8000
    align_next: 0x8000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size:  0x8000

prj,conf

CONFIG_I2C=y
CONFIG_SENSOR=y
CONFIG_PM_DEVICE=y
CONFIG_CRC=y
CONFIG_SERIAL=y
CONFIG_UART_ASYNC_API=y
CONFIG_UART_INTERRUPT_DRIVEN=y
CONFIG_HW_ID_LIBRARY=y

# Newlib
CONFIG_NEWLIB_LIBC=y

CONFIG_HEAP_MEM_POOL_SIZE=4096
CONFIG_MAIN_STACK_SIZE=4096

CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y

# SPI Flash
CONFIG_SPI=y
CONFIG_SPI_NOR=y
CONFIG_NORDIC_QSPI_NOR=n
# Enable flash operations.
CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_FLASH_PAGE_LAYOUT=y


CONFIG_REBOOT=y 

# Image manager
CONFIG_IMG_MANAGER=y
CONFIG_IMG_ERASE_PROGRESSIVELY=y

mcuboot.conf

# Example of sample specific Kconfig changes when building sample with MCUboot
# when using sysbuild.
CONFIG_MCUBOOT_LOG_LEVEL_WRN=y
CONFIG_MCUBOOT_DOWNGRADE_PREVENTION=y
CONFIG_MCUBOOT_LOG_LEVEL_DEFAULT=y
CONFIG_MCUBOOT_LOG_LEVEL_INF=y
CONFIG_BOOT_MAX_IMG_SECTORS=256
# CONFIG_FLASH=y
# CONFIG_MULTITHREADING=y
# CONFIG_SPI=y
# CONFIG_SPI_NOR=y
# CONFIG_SPI_NOR_FLASH_LAYOUT_PAGE_SIZE=4096

# # Enable flash operations

# CONFIG_PM_OVERRIDE_EXTERNAL_DRIVER_CHECK=y
# CONFIG_PM_EXTERNAL_FLASH_MCUBOOT_SECONDARY=y
# CONFIG_NORDIC_QSPI_NOR=n

sysbuild.conf
# Sysbuild configuration file.

# Enable MCUboot per default for this sample.
SB_CONFIG_BOOTLOADER_MCUBOOT=y
#SB_CONFIG_SECURE_BOOT_APPCORE=y


zephyr generation partitions.yml 
EMPTY_0:
  address: 0xf000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x1000
EMPTY_1:
  address: 0xe0000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_secondary
  region: flash_primary
  size: 0x20000
EMPTY_2:
  address: 0xc2000
  end_address: 0xc8000
  placement:
    after:
    - tfm_its
  region: flash_primary
  size: 0x6000
app:
  address: 0x68000
  end_address: 0xc0000
  region: flash_primary
  size: 0x58000
external_flash:
  address: 0x0
  end_address: 0x800000
  region: external_flash
  size: 0x800000
mcuboot:
  address: 0x0
  end_address: 0xf000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xf000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x8000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0xd8000
  orig_span: &id001
  - tfm
  - mcuboot_pad
  - app
  region: flash_primary
  sharers: 0x1
  size: 0xc8000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0xd8000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0xc7e00
  span: *id002
mcuboot_secondary:
  address: 0xd8000
  end_address: 0xe0000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x8000
    align_next: 0x8000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size: 0x8000
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20016000
  orig_span: &id003
  - tfm_sram
  region: sram_primary
  size: 0x16000
  span: *id003
nrf_modem_lib_ctrl:
  address: 0x20016000
  end_address: 0x200164e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - tfm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_rx:
  address: 0x20018568
  end_address: 0x2001a568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
nrf_modem_lib_sram:
  address: 0x20016000
  end_address: 0x2001a568
  orig_span: &id004
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x4568
  span: *id004
nrf_modem_lib_tx:
  address: 0x200164e8
  end_address: 0x20018568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2080
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
sram_nonsecure:
  address: 0x20016000
  end_address: 0x20040000
  orig_span: &id005
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x2a000
  span: *id005
sram_primary:
  address: 0x2001a568
  end_address: 0x20040000
  region: sram_primary
  size: 0x25a98
sram_secure:
  address: 0x20000000
  end_address: 0x20016000
  orig_span: &id006
  - tfm_sram
  region: sram_primary
  size: 0x16000
  span: *id006
tfm:
  address: 0x10200
  end_address: 0x68000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x57e00
tfm_its:
  address: 0xc0000
  end_address: 0xc2000
  inside:
  - tfm_storage
  placement:
    align:
      start: 0x8000
    before:
    - tfm_otp_nv_counters
  region: flash_primary
  size: 0x2000
tfm_nonsecure:
  address: 0x68000
  end_address: 0xd8000
  orig_span: &id007
  - app
  region: flash_primary
  size: 0x70000
  span: *id007
tfm_otp_nv_counters:
  address: 0xc8000
  end_address: 0xca000
  inside:
  - tfm_storage
  placement:
    align:
      start: 0x8000
    before:
    - tfm_ps
  region: flash_primary
  size: 0x2000
tfm_ps:
  address: 0xd0000
  end_address: 0xd4000
  inside:
  - tfm_storage
  placement:
    align:
      start: 0x8000
    before:
    - end
  region: flash_primary
  size: 0x4000
tfm_secure:
  address: 0x10000
  end_address: 0x68000
  orig_span: &id008
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x58000
  span: *id008
tfm_sram:
  address: 0x20000000
  end_address: 0x20016000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x16000
tfm_storage:
  address: 0xc0000
  end_address: 0xd4000
  orig_span: &id009
  - tfm_ps
  - tfm_its
  - tfm_otp_nv_counters
  region: flash_primary
  size: 0x14000
  span: *id009

The output 


And the funny thing is sometimes it boots into application but after hours of waiting here. As I left it like this yesterday and the application was booted and I again restarted the mcu the application got stuck here


Here is my project folder setup



and here are my project based .conf files which have encryption and other stuff activated
C_PLUS_PLUS.conf

CONFIG_CPP=y
CONFIG_REQUIRES_FULL_LIBCPP=y
CONFIG_STD_CPP20=y
CONFIG_NEWLIB_LIBC=y


crypto.conf
# ---------- TF-M integration ----------
CONFIG_TFM_PROFILE_TYPE_NOT_SET=y     # pick secure services by hand

# ---------- nRF Security + PSA layer ----------
CONFIG_NRF_SECURITY=y
CONFIG_MBEDTLS_PSA_CRYPTO_C=y

# ---------- use CryptoCell hardware driver ----------
CONFIG_PSA_CRYPTO_DRIVER_CC3XX=y
CONFIG_PSA_CRYPTO_DRIVER_OBERON=y

# ---------- algorithms & key-types  ----------
CONFIG_PSA_WANT_ALG_ECDH=y
CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_GENERATE=y
CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_IMPORT=y
CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_EXPORT=y
#CONFIG_PSA_WANT_ECC_CURVE_CURVE25519=y
#following gives Curve25519 support
CONFIG_PSA_WANT_ECC_MONTGOMERY_255=y

#---------- key-derivation & hash for post-ECDH ----------
CONFIG_PSA_WANT_ALG_HKDF=y
CONFIG_PSA_WANT_ALG_SHA_256=y
CONFIG_PSA_WANT_ALG_HMAC=y
CONFIG_PSA_WANT_KEY_TYPE_HMAC=y
#----------- CHACHA20 poly1305 -------------
CONFIG_PSA_WANT_KEY_TYPE_CHACHA20=y
CONFIG_PSA_WANT_ALG_CHACHA20_POLY1305=y

# ---------- entropy source for key generation ----------
CONFIG_PSA_WANT_GENERATE_RANDOM=y

logging.conf
CONFIG_LOG=y
CONFIG_LOG_BUFFER_SIZE=4096
CONFIG_LOG_BACKEND_UART=y   
CONFIG_SHELL_LOG_BACKEND=n    # ← disable log-to-shell



# --- Shell core -------------------------------------------------
CONFIG_SHELL=y
CONFIG_SHELL_STACK_SIZE=5120
CONFIG_SHELL_PROMPT_UART="gasSensor: "
CONFIG_SHELL_CMD_ROOT="gasSensor"
CONFIG_SHELL_BACKEND_SERIAL=y
#CONFIG_SHELL_PROMPT="gas> "          # prompt string

# --- turn OFF all built-in command groups --------------------------
CONFIG_SHELL_CMDS=n           # generic cmds: clear, history, resize, retval
CONFIG_DEVICE_SHELL=n         # device, devmem
CONFIG_DEVMEM_SHELL=n
CONFIG_KERNEL_SHELL=n         # kernel timing, stacks, uptime, ...

# (optional) keep command history?
CONFIG_SHELL_HISTORY=n        # turn off if you don’t need ↑ key recall


lte.conf
CONFIG_NRF_MODEM_LIB=y
CONFIG_LTE_LINK_CONTROL=y
CONFIG_LTE_NETWORK_MODE_LTE_M_NBIOT=y

#ONFIG_AT_HOST_LIBRARY=y


#power saving
CONFIG_LTE_LC_EDRX_MODULE=y
CONFIG_LTE_LC_PSM_MODULE=y
CONFIG_LTE_PSM_REQ=y
CONFIG_LTE_EDRX_REQ=y



mqtt,networking and tls conf
# MQTT
CONFIG_MQTT_LIB=y
CONFIG_MQTT_CLEAN_SESSION=y

# Networking
CONFIG_NETWORKING=y
CONFIG_NET_NATIVE=n
CONFIG_NET_SOCKETS_OFFLOAD=y
CONFIG_NET_SOCKETS=y
CONFIG_POSIX_API=y

CONFIG_MODEM_KEY_MGMT=y

CONFIG_MQTT_LIB_TLS=y




I will get back to external flash mcuboot issue later but first I want to boot into my normal application as a normal bootloader

Parents
  • Hi,

    The size of the secondary mcuboot partition has me a bit interested.

    The size is 0x8000 in both your defined pm_static and the generated partitioning map, whereas it should be the same size as your primary mcuboot application slot.

    I think that maybe this course on our academy pages with theory for DFU in NCS could be helpful https://academy.nordicsemi.com/courses/nrf-connect-sdk-intermediate/lessons/lesson-9-bootloaders-and-dfu-fota/ 

    There's lessons (with solutions) for your use case, i.e OTA with external flash and a lesson for LTE OTA (if that's the end goal) as well as BLE OTA.

    Let me know if you've reviewed this topic on the pages already or not

    Kind regards,
    Andreas

  • yes I gave that size because right now I dont want to load any application to secondary slot.
    I only want my system to load into my application thats all, basically I just want right now the mcuboot to jump to my main application and I think it should as it wont get any header on secondary slot and it should jump to primary.

    Thats the reason I kept secondary slot that low memory footprint.

    As you can see the mcuboot says it jumps and I also mentioned that I left it like this yesterday when I came back to it in the morning surprisingly my application was working


  • okay so after going through some resources and also following nrf connect intermediate guide I was able to use external flash as an secondary storage


    This auto generated parition.yml file


    As you can see now in output, all the warning are not even showing and mcuboot is showing correct outputs now but still when its jump to first image slot it get stuck and my application is never booted


    After testing this I implemented my own static_pm.yml file

    #include <autoconf.h>
    EMPTY_0:
      address: 0xF000
      end_address: 0x10000
      placement:
        before:
        - mcuboot_pad
      region: flash_primary
      size: 0x1000
    app:
      address: 0x68000
      end_address: 0xD8000
      region: flash_primary
      size: 0x70000
    mcuboot:
      address: 0x0
      end_address: 0xF000
      placement:
        before:
        - mcuboot_primary
      region: flash_primary
      size: 0xF000
    mcuboot_pad:
      address: 0x10000
      end_address: 0x10200
      placement:
        align:
          start: 0x8000
        before:
        - mcuboot_primary_app
      region: flash_primary
      size: 0x200
    mcuboot_primary:
      address: 0x10000
      end_address: 0xD8000
      orig_span: &id001
      - tfm
      - mcuboot_pad
      - app
      region: flash_primary
      sharers: 0x1
      size: 0xC8000
      span: *id001
    mcuboot_primary_app:
      address: 0x10200
      end_address: 0xD8000
      orig_span: &id002
      - app
      - tfm
      region: flash_primary
      size: 0xC7E00
      span: *id002
    otp:
      address: 0xff8108
      end_address: 0xff83fc
      region: otp
      size: 0x2f4
    tfm:
      address: 0x10200
      end_address: 0x68000
      inside:
      - mcuboot_primary_app
      placement:
        before:
        - app
      region: flash_primary
      size: 0x57E00
    tfm_secure:
      address: 0x10000
      end_address: 0x68000
      orig_span: &id008
      - mcuboot_pad
      - tfm
      region: flash_primary
      size: 0x58000
      span: *id008
    tfm_nonsecure:
      address: 0x68000
      end_address: 0xD8000
      orig_span: &id007
      - app
      region: flash_primary
      size: 0x70000
      span: *id007
    external_flash:
      address: 0x0
      device: mx25r64
      end_address: 0x800000
      region: external_flash
      size: 0x800000
    mcuboot_secondary:
      address: 0x0
      device: mx25r64
      region: external_flash
      size: 0xC8000


    The same response with this one 

    So there is definitely something going on with my application code I feel like that does not let mcuboot jump to it

    what are your speculation ?
  • my hunch is that, when the issue is with following crypto.conf file in my project

    # ---------- TF-M integration ----------
    CONFIG_TFM_PROFILE_TYPE_NOT_SET=y     # pick secure services by hand
    
    # ---------- nRF Security + PSA layer ----------
    CONFIG_NRF_SECURITY=y
    CONFIG_MBEDTLS_PSA_CRYPTO_C=y
    
    # ---------- use CryptoCell hardware driver ----------
    CONFIG_PSA_CRYPTO_DRIVER_CC3XX=y
    CONFIG_PSA_CRYPTO_DRIVER_OBERON=y
    
    # ---------- algorithms & key-types  ----------
    CONFIG_PSA_WANT_ALG_ECDH=y
    CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_GENERATE=y
    CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_IMPORT=y
    CONFIG_PSA_WANT_KEY_TYPE_ECC_KEY_PAIR_EXPORT=y
    #CONFIG_PSA_WANT_ECC_CURVE_CURVE25519=y
    #following gives Curve25519 support
    CONFIG_PSA_WANT_ECC_MONTGOMERY_255=y
    
    #---------- key-derivation & hash for post-ECDH ----------
    CONFIG_PSA_WANT_ALG_HKDF=y
    CONFIG_PSA_WANT_ALG_SHA_256=y
    CONFIG_PSA_WANT_ALG_HMAC=y
    CONFIG_PSA_WANT_KEY_TYPE_HMAC=y
    #----------- CHACHA20 poly1305 -------------
    CONFIG_PSA_WANT_KEY_TYPE_CHACHA20=y
    CONFIG_PSA_WANT_ALG_CHACHA20_POLY1305=y
    
    # ---------- entropy source for key generation ----------
    CONFIG_PSA_WANT_GENERATE_RANDOM=y
    


    maybe CONFIG_TFM_PROFILE_TYPE_NOT_SET=y 

    it is not sitting well with mcuboot ?

  • Also I tried external flash with hello_world program and my system immediately booted into application


    Also I added my above crypto.conf to project and hello_world application still booted



    Also I kept same partitions for both projects hello_world and my main application



    I am using external flash in my application as well to store certificates may that could be an issue ?? I kept address space different ofcourse but still



    Inside this library I am using normal flash_write,erase and read functions


    BTW here is my code https://github.com/haseebzaib/GasSensor_Device
    maybe will be easier to test on your side as well

  • Hi,

    Haseeb Zaib said:
    As you can see now in output, all the warning are not even showing and mcuboot is showing correct outputs now but still when its jump to first image slot it get stuck and my application is never booted

    Thats good, then we've removed one uncertainty from the equation. Based on what you're describing it seems to me that this is not caused by the bootloader and that there may be some softlock in app start, i.e the fault is isolated to the application

    Here's some suggestions:

    • Verify that the address that MCUboot attempts to boot does indeed match the start address of the application
    • Could you remove MCUboot and attempt to just have the application (and potentially certificates on external flash) and see if this has the same behavior? If so, then we've isolated that the issue is in the application.
    • The next thing is to check if it's because of the certificates on the external flash, i.e have no bootloader + certificates on internal flash

    Kind regards,
    Andreas

    • Could you remove MCUboot and attempt to just have the application (and potentially certificates on external flash) and see if this has the same behavior? If so, then we've isolated that the issue is in the application.
    • The next thing is to check if it's because of the certificates on the external flash, i.e have no bootloader + certificates on internal flash

    sorry I dont get these point of yours

    Could you remove MCUboot and attempt to just have the application (and potentially certificates on external flash) and see if this has the same behavior? If so, then we've isolated that the issue is in the application.

    The application works fine without mcuboot and boots properly, as I have already tested complete application with certs in external flash

    The next thing is to check if it's because of the certificates on the external flash, i.e have no bootloader + certificates on internal flash

    I dont get it I think you meant I keep certs in internal flash and then try mcuboot right ?

Reply
    • Could you remove MCUboot and attempt to just have the application (and potentially certificates on external flash) and see if this has the same behavior? If so, then we've isolated that the issue is in the application.
    • The next thing is to check if it's because of the certificates on the external flash, i.e have no bootloader + certificates on internal flash

    sorry I dont get these point of yours

    Could you remove MCUboot and attempt to just have the application (and potentially certificates on external flash) and see if this has the same behavior? If so, then we've isolated that the issue is in the application.

    The application works fine without mcuboot and boots properly, as I have already tested complete application with certs in external flash

    The next thing is to check if it's because of the certificates on the external flash, i.e have no bootloader + certificates on internal flash

    I dont get it I think you meant I keep certs in internal flash and then try mcuboot right ?

Children
No Data
Related