Debug external_flash and nonsecure_storage partitions in pm_static.yml (external SPI flash)

[caeg]

My project is using NCS 2.6.0, builds the _ns variant with TF-M and mcuboot, and targets nRF5340 with an 8MB QSPI flash (it's a custom board, but mirrors the 5340DK in this regard).

For some time now I've operated with a certain partition layout. In this arrangement the SPI flash contains 0xf0000 for app core OTA, 0x40000 for net core OTA, 0x4000 for settings NVS, and 0xa000 for lfs. The remaining 0x6c2000 of the flash is allocated to the external_flash partition. This is the pm_static.yml I'm starting from:

mcuboot_secondary:
  address: 0x0
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0xf0000
  placement:
    align:
      start: 0x4
  region: external_flash
  share_size:
  - mcuboot_primary
  size: 0xf0000
mcuboot_secondary_1:
  address: 0xf0000
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x130000
  region: external_flash
  size: 0x40000
external_flash:
  address: 0x13e000
  end_address: 0x800000
  region: external_flash
  size: 0x6c2000
nonsecure_storage:
  address: 0x130000
  end_address: 0x13e000
  orig_span: &id008
  - settings_storage
  - littlefs_storage
  region: external_flash
  size: 0xe000
  span: *id008
settings_storage:
  address: 0x130000
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x134000
  inside:
  - nonsecure_storage
  placement:
    align:
      start: 0x4000
    before:
    - littlefs_storage
  region: external_flash
  size: 0x4000
littlefs_storage:
  address: 0x134000
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x13e000
  placement:
    before:
    - end
  region: external_flash
  size: 0xa000
mcuboot:
  address: 0x0
  end_address: 0x10000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0x10000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x4000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0x100000
  orig_span: &id001
  - app
  - tfm
  - mcuboot_pad
  region: flash_primary
  size: 0xf0000
  span: *id001
mcuboot_primary_1:
  address: 0x0
  device: nordic_ram_flash_controller
  end_address: 0x40000
  region: ram_flash
  size: 0x40000
mcuboot_primary_app:
  address: 0x10200
  end_address: 0x100000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0xefe00
  span: *id002
app:
  address: 0x20000
  end_address: 0x100000
  region: flash_primary
  size: 0xf0000
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20080000
  orig_span: &id003
  - sram_primary
  - rpmsg_nrf53_sram
  - tfm_sram
  - pcd_sram
  region: sram_primary
  size: 0x80000
  span: *id003
otp:
  address: 0xff8100
  end_address: 0xff83fc
  region: otp
  size: 0x2fc
pcd_sram:
  address: 0x20000000
  end_address: 0x20002000
  placement:
    before:
    - tfm_sram
  region: sram_primary
  size: 0x2000
ram_flash:
  address: 0x40000
  end_address: 0x40000
  region: ram_flash
  size: 0x0
rpmsg_nrf53_sram:
  address: 0x20070000
  end_address: 0x20080000
  placement:
    before:
    - end
  region: sram_primary
  size: 0x10000
sram_nonsecure:
  address: 0x2000a000
  end_address: 0x20080000
  orig_span: &id004
  - sram_primary
  - rpmsg_nrf53_sram
  region: sram_primary
  size: 0x76000
  span: *id004
sram_primary:
  address: 0x2000a000
  end_address: 0x20070000
  region: sram_primary
  size: 0x66000
sram_secure:
  address: 0x20000000
  end_address: 0x2000a000
  orig_span: &id005
  - pcd_sram
  - tfm_sram
  region: sram_primary
  size: 0xa000
  span: *id005
tfm:
  address: 0x10200
  end_address: 0x20000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0xfe00
tfm_nonsecure:
  address: 0x20000
  end_address: 0x100000
  orig_span: &id006
  - app
  region: flash_primary
  size: 0xe0000
  span: *id006
tfm_secure:
  address: 0x10000
  end_address: 0x20000
  orig_span: &id007
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x10000
  span: *id007
tfm_sram:
  address: 0x20002000
  end_address: 0x2000a000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x8000

It's confusing that there is a region called external_flash and also a partition called external_flash. I don't recall how this came to be -- most likely is that it was generated this way in build/partitions.yml which I copied when originally creating pm_static.yml. I note that the file v2.6.0/nrf/samples/nrf5340/extxip_smp_svr/pm_static.yml also contains a partition external_flash, but in this sample, their external_flash partition overlaps their mcuboot_primary_2 and mcuboot_secondary_2 partitions. So here's one problem: my understanding of the purpose of the external_flash partition is flawed; I had though it should be covering all of, and exclusively, the unused portion of flash. I deleted this partition in my changes below, but I question whether I should be keeping it and what addresses it should be assigned.

I'd like to add 0x10000 pad between mcuboot_secondary and mcuboot_secondary_1, grow the nonsecure_storage to 0x20000, and allocate the remaining 0x6a0000 of the flash to lfs. I modified the pm_static.yml accordingly: insert secondary_pad, slide mcuboot_secondary_1, adjust the boundaries on nonsecure_storage, slide and adjust settings_storage and littlefs_storage.

mcuboot_secondary:
  address: 0x0
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0xf0000
  placement:
    align:
      start: 0x4
  region: external_flash
  share_size:
  - mcuboot_primary
  size: 0xf0000
secondary_pad: # Pad out the app image space to a full 1 MB
  address: 0xf0000
  end_address: 0x100000
  region: external_flash
  size: 0x10000
mcuboot_secondary_1:
  address: 0x100000
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x140000
  region: external_flash
  size: 0x40000
nonsecure_storage:
  address: 0x140000
  end_address: 0x800000
  orig_span: &id008
  - settings_storage
  - littlefs_storage
  region: external_flash
  size: 0x6c0000
  span: *id008
settings_storage:
  address: 0x140000
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x160000
  inside:
  - nonsecure_storage
  placement:
    align:
      start: 0x10000
    before:
    - littlefs_storage
  region: external_flash
  size: 0x20000
littlefs_storage:
  address: 0x160000
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x800000
  placement:
    before:
    - end
  region: external_flash
  size: 0x6a0000
app:
  address: 0x20000
  end_address: 0x100000
  region: flash_primary
  size: 0xf0000
mcuboot:
  address: 0x0
  end_address: 0x10000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0x10000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x4000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0x100000
  orig_span: &id001
  - app
  - tfm
  - mcuboot_pad
  region: flash_primary
  size: 0xf0000
  span: *id001
mcuboot_primary_1:
  address: 0x0
  device: nordic_ram_flash_controller
  end_address: 0x40000
  region: ram_flash
  size: 0x40000
mcuboot_primary_app:
  address: 0x10200
  end_address: 0x100000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0xefe00
  span: *id002
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20080000
  orig_span: &id003
  - sram_primary
  - rpmsg_nrf53_sram
  - tfm_sram
  - pcd_sram
  region: sram_primary
  size: 0x80000
  span: *id003
otp:
  address: 0xff8100
  end_address: 0xff83fc
  region: otp
  size: 0x2fc
pcd_sram:
  address: 0x20000000
  end_address: 0x20002000
  placement:
    before:
    - tfm_sram
  region: sram_primary
  size: 0x2000
ram_flash:
  address: 0x40000
  end_address: 0x40000
  region: ram_flash
  size: 0x0
rpmsg_nrf53_sram:
  address: 0x20070000
  end_address: 0x20080000
  placement:
    before:
    - end
  region: sram_primary
  size: 0x10000
sram_nonsecure:
  address: 0x2000a000
  end_address: 0x20080000
  orig_span: &id004
  - sram_primary
  - rpmsg_nrf53_sram
  region: sram_primary
  size: 0x76000
  span: *id004
sram_primary:
  address: 0x2000a000
  end_address: 0x20070000
  region: sram_primary
  size: 0x66000
sram_secure:
  address: 0x20000000
  end_address: 0x2000a000
  orig_span: &id005
  - pcd_sram
  - tfm_sram
  region: sram_primary
  size: 0xa000
  span: *id005
tfm:
  address: 0x10200
  end_address: 0x20000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0xfe00
tfm_nonsecure:
  address: 0x20000
  end_address: 0x100000
  orig_span: &id006
  - app
  region: flash_primary
  size: 0xe0000
  span: *id006
tfm_secure:
  address: 0x10000
  end_address: 0x20000
  orig_span: &id007
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x10000
  span: *id007
tfm_sram:
  address: 0x20002000
  end_address: 0x2000a000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x8000

Unfortunately this results in an image that doesn't boot. With some mcuboot serial/uart enabled in the serial console I see mcuboot starting repeatedly. Probably my app firmware is starting and something is having a panic during its initialization and causing a reboot but I haven't pursued that.

�I: Starting bootloader image slot
I: Image index: 0, Swap type: none
I: Image index: 1, Swap type: none
I: Bootloader chainload address offset: 0x10000
�I: Starting bootloader image slot
I: Image index: 0, Swap type: none
I: Image index: 1, Swap type: none
I: Bootloader chainload address offset: 0x10000
�I: Starting bootloader image slot
I: Image index: 0, Swap type: none
I: Image index: 1, Swap type: none
I: Bootloader chainload address offset: 0x10000

After quite a bit of iteration I discovered that if I manipulate the nonsecure_storage boundaries to be more similar to those I had originally, then the image boots and appears to run fine. Additionally I've tried erasing the QSPI flash with nrfjprog.exe --qspieraseall just in case there could be data from the prior partitioning that upset some module in my firmware - that didn't help.

mcuboot_secondary:
  address: 0x0
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0xf0000
  placement:
    align:
      start: 0x4
  region: external_flash
  share_size:
  - mcuboot_primary
  size: 0xf0000
secondary_pad: # Pad out the app image space to a full 1 MB
  address: 0xf0000
  end_address: 0x100000
  region: external_flash
  size: 0x10000
mcuboot_secondary_1:
  address: 0x100000
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x140000
  region: external_flash
  size: 0x40000
nonsecure_storage:
  address: 0x140000
  end_address: 0x14e000
  orig_span: &id008
  - settings_storage
  - littlefs_storage
  region: external_flash
  size: 0xe000
  span: *id008
settings_storage:
  address: 0x140000
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x160000
  inside:
  - nonsecure_storage
  placement:
    align:
      start: 0x10000
    before:
    - littlefs_storage
  region: external_flash
  size: 0x20000
littlefs_storage:
  address: 0x160000
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x800000
  placement:
    before:
    - end
  region: external_flash
  size: 0x6a0000
app:
  address: 0x20000
  end_address: 0x100000
  region: flash_primary
  size: 0xf0000
mcuboot:
  address: 0x0
  end_address: 0x10000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0x10000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x4000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0x100000
  orig_span: &id001
  - app
  - tfm
  - mcuboot_pad
  region: flash_primary
  size: 0xf0000
  span: *id001
mcuboot_primary_1:
  address: 0x0
  device: nordic_ram_flash_controller
  end_address: 0x40000
  region: ram_flash
  size: 0x40000
mcuboot_primary_app:
  address: 0x10200
  end_address: 0x100000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0xefe00
  span: *id002
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20080000
  orig_span: &id003
  - sram_primary
  - rpmsg_nrf53_sram
  - tfm_sram
  - pcd_sram
  region: sram_primary
  size: 0x80000
  span: *id003
otp:
  address: 0xff8100
  end_address: 0xff83fc
  region: otp
  size: 0x2fc
pcd_sram:
  address: 0x20000000
  end_address: 0x20002000
  placement:
    before:
    - tfm_sram
  region: sram_primary
  size: 0x2000
ram_flash:
  address: 0x40000
  end_address: 0x40000
  region: ram_flash
  size: 0x0
rpmsg_nrf53_sram:
  address: 0x20070000
  end_address: 0x20080000
  placement:
    before:
    - end
  region: sram_primary
  size: 0x10000
sram_nonsecure:
  address: 0x2000a000
  end_address: 0x20080000
  orig_span: &id004
  - sram_primary
  - rpmsg_nrf53_sram
  region: sram_primary
  size: 0x76000
  span: *id004
sram_primary:
  address: 0x2000a000
  end_address: 0x20070000
  region: sram_primary
  size: 0x66000
sram_secure:
  address: 0x20000000
  end_address: 0x2000a000
  orig_span: &id005
  - pcd_sram
  - tfm_sram
  region: sram_primary
  size: 0xa000
  span: *id005
tfm:
  address: 0x10200
  end_address: 0x20000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0xfe00
tfm_nonsecure:
  address: 0x20000
  end_address: 0x100000
  orig_span: &id006
  - app
  region: flash_primary
  size: 0xe0000
  span: *id006
tfm_secure:
  address: 0x10000
  end_address: 0x20000
  orig_span: &id007
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x10000
  span: *id007
tfm_sram:
  address: 0x20002000
  end_address: 0x2000a000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x8000

Could you help me figure out why adjusting the nonsecure_storage addresses so that they do not properly span settings_storage and littlefs_storage actually results in an image that boots? Also, if you have any insight about whether the partition external_flash should stay or go, that would be appreciated too.

[Vidar Berg]

Hello,

You can view the 'external_flash' partition as a placeholder covering the unused are of the external flash region, but I agree that the naming could have been more descriptive. When you let the PM allocate the partitions dynamically by not using a static partition file, it will place new partitions into the 'external_flash' region and reduce the size of the 'external_flash' partition accordingly. For instance, when enabling the CONFIG_PM_PARTITION_REGION_LITTLEFS_EXTERNAL=y symbol to place the FS in the external flash region.

So here's one problem: my understanding of the purpose of the external_flash partition is flawed; I had though it should be covering all of, and exclusively, the unused portion of flash. I deleted this partition in my changes below, but I question whether I should be keeping it and what addresses it should be assigned.

Strange, I'm seeing the behaviour you expected when building the smp_srv sample with the QSPI flash device:

Test sample:

smp_srv_ext_flash.zip

Here is the partition file from my build for comparison:

EMPTY_0:
  address: 0xe000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x2000
app:
  address: 0x18000
  end_address: 0x100000
  region: flash_primary
  size: 0xe8000
external_flash:
  address: 0x100000
  end_address: 0x800000
  region: external_flash
  size: 0x700000
littlefs_storage:
  address: 0xf0000
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x100000
  placement:
    before:
    - tfm_storage
    - end
  region: external_flash
  size: 0x10000
mcuboot:
  address: 0x0
  end_address: 0xe000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xe000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x4000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0x100000
  orig_span: &id001
  - mcuboot_pad
  - tfm
  - app
  region: flash_primary
  size: 0xf0000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0x100000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0xefe00
  span: *id002
mcuboot_secondary:
  address: 0x0
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0xf0000
  placement:
    align:
      start: 0x4
  region: external_flash
  share_size:
  - mcuboot_primary
  size: 0xf0000
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20080000
  orig_span: &id003
  - tfm_sram
  - sram_primary
  - rpmsg_nrf53_sram
  region: sram_primary
  size: 0x80000
  span: *id003
otp:
  address: 0xff8100
  end_address: 0xff83fc
  region: otp
  size: 0x2fc
rpmsg_nrf53_sram:
  address: 0x20070000
  end_address: 0x20080000
  placement:
    before:
    - end
  region: sram_primary
  size: 0x10000
sram_nonsecure:
  address: 0x20008000
  end_address: 0x20080000
  orig_span: &id004
  - sram_primary
  - rpmsg_nrf53_sram
  region: sram_primary
  size: 0x78000
  span: *id004
sram_primary:
  address: 0x20008000
  end_address: 0x20070000
  region: sram_primary
  size: 0x68000
sram_secure:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id005
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id005
tfm:
  address: 0x10200
  end_address: 0x18000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x7e00
tfm_nonsecure:
  address: 0x18000
  end_address: 0x100000
  orig_span: &id006
  - app
  region: flash_primary
  size: 0xe8000
  span: *id006
tfm_secure:
  address: 0x10000
  end_address: 0x18000
  orig_span: &id007
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x8000
  span: *id007
tfm_sram:
  address: 0x20000000
  end_address: 0x20008000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x8000

It looks like the key difference is that I have a 'tfm' partition which ensures the TF-M application is placed in the tfm-secure region of flash.

Best regards,

Vidar

[caeg]

My files have a tfm partition also, so that's not a difference in yours and mine. But I see that yours entirely omits the nonsecure_storage partition. My pm_static.yml was derived some time ago when I was working with SDK 2.3.0. Since then we've updated to 2.6.0. I got curious what partition manager in 2.6.0 might generate, so I added the items below in prj.conf, removed my pm_static.yml, and ran a build.

CONFIG_PM_PARTITION_SIZE_LITTLEFS=0x6a0000
CONFIG_PM_PARTITION_REGION_LITTLEFS_EXTERNAL=y
CONFIG_PM_PARTITION_SIZE_SETTINGS_STORAGE=0x20000
CONFIG_PM_PARTITION_REGION_SETTINGS_STORAGE_EXTERNAL=y

Like yours, my generated partitions.yml did not have any nonsecure_storage partition. And it booted and ran fine. I tried restoring my problematic pm_static.yml file above and deleting this partition from it; this build also works.

Maybe the core problem is simply that nonsecure_storage partition is no longer applicable. How would I confirm this?

[Vidar Berg]

Sorry, I'm not sure how I missed the TFM partition while reviewing your partition file. Another thing I overlooked is that you had the nonsecure_storage area in external flash - it should only be added if you have nonsecure storage partitions in internal flash, as the SPU — System protection unit does not cover externally mapped memory. Using it for external flash will result in an erroneous SPU configuration. This probably lead to a fault exception in your TF-M app.

secure and non-secure area in internal flash