nRF internal flash utilized for MCUboot, external for FS

[kuchx]

Hello,

I been struggling with this issue for a couple of days and can't manage to find the reason behind it. 

About the project:

We are developing a IoT device firmware based on nRF9160 SiP. We are using first-stage immutable bootloader (MCUboot) and also LittleFS file system. System was working fine whilst it was running only on internal flash. Once we enabled external flash, we dedicated all of its memory for file logging. Since this is a multi-image build the Partition Manager is used to generate flash partitions. At the beginning the PM created MCUboot secondary image on external flash, because `CONFIG_PM_EXTERNAL_FLASH_MCUBOOT_SECONDARY=n`by default was set to y but we changed that. At this point everything seems to be okay - MCUboot both partitions are on internal flash but filesystem is on external

The issue:

For over-the-air upgrade we use block-wise transfer and receive FW chunk by chunk from server and saved the chunk to secondary mcuboot_image slot using `flash_img_buffered_write`. After enabling external flash, flash_img_buffered_write causes secure fault which causes the system to reset. With default `partitions.yml` generated file the fault is on the first call. When I tried to add `pm_static.yml` file at some point the fault happened later but the fault still existed. 

What have I tried:

I've tried creating and playing around with `pm_static.yml` file partition properties and at some point even placements but didn't manage to make it work. I also tried to replicate partitions as it was when the system worked only on internal flash just moved the littlefs_storage to external_flash partition. I have checked that when partition context passed to flash_img_buffered_write is with correct ID and address, `pm_static.yml` partition properties seems to be logical. With addr2line.exe returns that the faulting line is at `nrf_nvmc.h:378` which is a call to write a word. I tried to find a similar issue in this forum but without luck. I believe the issue is because of incorrect partitioning because at some point while I was playing with some partition properties the Secure fault was introduced only later - after couple of chunks was already saved in flash not on the first call as it is most of the time

Resources:

Here is the generated partitions.yml file which out of the box was causing secure fault on first write call

EMPTY_0:
  address: 0xc000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x4000
app:
  address: 0x18000
  end_address: 0x88000
  region: flash_primary
  size: 0x70000
external_flash:
  address: 0x6000
  end_address: 0x400000
  region: external_flash
  size: 0x3fa000
littlefs_storage:
  address: 0x0
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x6000
  placement:
    before:
    - tfm_storage
    - end
  region: external_flash
  size: 0x6000
mcuboot:
  address: 0x0
  end_address: 0xc000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xc000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x8000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0x88000
  orig_span: &id001
  - app
  - tfm
  - mcuboot_pad
  region: flash_primary
  sharers: 0x1
  size: 0x78000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0x88000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0x77e00
  span: *id002
mcuboot_secondary:
  address: 0x88000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x8000
    align_next: 0x8000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size: 0x78000
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id003
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id003
nrf_modem_lib_ctrl:
  address: 0x20008000
  end_address: 0x200084e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - tfm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_rx:
  address: 0x2000a568
  end_address: 0x2000c568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
nrf_modem_lib_sram:
  address: 0x20008000
  end_address: 0x2000c568
  orig_span: &id004
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x4568
  span: *id004
nrf_modem_lib_tx:
  address: 0x200084e8
  end_address: 0x2000a568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2080
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
sram_nonsecure:
  address: 0x20008000
  end_address: 0x20040000
  orig_span: &id005
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x38000
  span: *id005
sram_primary:
  address: 0x2000c568
  end_address: 0x20040000
  region: sram_primary
  size: 0x33a98
sram_secure:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id006
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id006
tfm:
  address: 0x10200
  end_address: 0x18000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x7e00
tfm_nonsecure:
  address: 0x18000
  end_address: 0x88000
  orig_span: &id007
  - app
  region: flash_primary
  size: 0x70000
  span: *id007
tfm_secure:
  address: 0x10000
  end_address: 0x18000
  orig_span: &id008
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x8000
  span: *id008
tfm_sram:
  address: 0x20000000
  end_address: 0x20008000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x8000

Here is my pm_static.yml file which I added based on `partitions.yml` output when everything was on internal flash.

EMPTY_0:
  address: 0xc000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x4000
EMPTY_1:
  address: 0xf0000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_secondary
  region: flash_primary
  size: 0x10000
app:
  address: 0x18000
  end_address: 0x80000
  region: flash_primary
  size: 0x68000
external_flash:
  address: 0x6000
  end_address: 0x800000
  region: external_flash
  size: 0x7fa000
littlefs_storage:
  address: 0x0
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x6000
  inside:
  - nonsecure_storage
  placement:
    before:
    - end
  region: external_flash
  size: 0x6000
mcuboot:
  address: 0x0
  end_address: 0xc000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xc000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x8000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0x80000
  orig_span: &id001
  - mcuboot_pad
  - tfm
  - app
  region: flash_primary
  sharers: 0x1
  size: 0x70000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0x80000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0x6fe00
  span: *id002
mcuboot_secondary:
  address: 0x80000
  end_address: 0xf0000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x8000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size: 0x70000
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id003
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id003
nonsecure_storage:
  address: 0x0
  end_address: 0x6000
  orig_span: &id004
  - littlefs_storage
  region: external_flash
  size: 0x6000
  span: *id004
nrf_modem_lib_ctrl:
  address: 0x20008000
  end_address: 0x200084e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - tfm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_rx:
  address: 0x2000a568
  end_address: 0x2000c568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
nrf_modem_lib_sram:
  address: 0x20008000
  end_address: 0x2000c568
  orig_span: &id005
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x4568
  span: *id005
nrf_modem_lib_tx:
  address: 0x200084e8
  end_address: 0x2000a568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2080
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
sram_nonsecure:
  address: 0x20008000
  end_address: 0x20040000
  orig_span: &id006
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x38000
  span: *id006
sram_primary:
  address: 0x2000c568
  end_address: 0x20040000
  region: sram_primary
  size: 0x33a98
sram_secure:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id007
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id007
tfm:
  address: 0x10200
  end_address: 0x18000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x7e00
tfm_nonsecure:
  address: 0x18000
  end_address: 0x80000
  orig_span: &id008
  - app
  region: flash_primary
  size: 0x68000
  span: *id008
tfm_secure:
  address: 0x10000
  end_address: 0x18000
  orig_span: &id009
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x8000
  span: *id009
tfm_sram:
  address: 0x20000000
  end_address: 0x20008000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x8000

Below is the output from the exception call if that helps with anything

[00:00:07.133,087] <err> os: ***** SECURE FAULT *****
[00:00:07.133,178] <err> os:   Address: 0x80000
[00:00:07.133,209] <err> os:   Attribution unit violation
[00:00:07.133,239] <err> os: r0/a1:  0x00080000  r1/a2:  0x96f3b83d  r2/a3:  0x40039000
[00:00:07.133,270] <err> os: r3/a4:  0x80000000 r12/ip:  0x2001aac4 r14/lr:  0x0002a55b
[00:00:07.133,300] <err> os:  xpsr:  0x81000000
[00:00:07.133,300] <err> os: Faulting instruction address (r15/pc): 0x0002a4a2
[00:00:07.133,422] <err> os: >>> ZEPHYR FATAL ERROR 41: Unknown error on CPU 0
[00:00:07.133,514] <err> os: Current thread: 0x2000d5b0 (unknown)

To me seems to be okay - app tries to write from non-secure region to non-secure region, the partition address's don't overlap, the partition context is correct.. Maybe somebody with a fresh pair of eyes can spot the issue right away. 

EDIT: Almost forgot to mention, I am developing on a custom board but the issue can be replicated on DK. Working with SDK V2.6.0 

Thanks,

Andris

[Vidar Berg]

Hello Andris,

It is not possible to map partitions as secure and non-secure in external flash as this only applies to internal flash. Therefore, please try removing the nonsecure_storage partition for the external memory. Tip: creating a partition report with the "west build -t partition_manager_report" from the command line makes it easier to see if the secure and non-secure regions are placed correctly.

Best regards,

Vidar

[kuchx]

Hi Vidar,

Thanks for the heads up. I removed the nonsecure_storage from pm_static.yml. But this does not have any effect on the issue I'm facing. As you can see in initially generated partitions.yml file (the first block of code) there is no such nonsecure_storage partition and the app still failed when trying to write to mcuboot_secondary partition!

Regards,

Andris

[Vidar Berg]

Hi Andris,

Yes, but in the first partition file you had the littlefs partition with this placement:

placement:
before:
- tfm_storage
- end

Can you create the partition manager report?

[kuchx]

Hi Vidar,

I updated the pm_static to this: 

EMPTY_0:
  address: 0xc000
  end_address: 0x10000
  placement:
    before:
    - mcuboot_pad
  region: flash_primary
  size: 0x4000
EMPTY_1:
  address: 0xf0000
  end_address: 0x100000
  placement:
    after:
    - mcuboot_secondary
  region: flash_primary
  size: 0x10000
app:
  address: 0x18000
  end_address: 0x80000
  region: flash_primary
  size: 0x68000
external_flash:
  address: 0x6000
  end_address: 0x800000
  region: external_flash
  size: 0x7fa000
littlefs_storage:
  address: 0x0
  device: DT_CHOSEN(nordic_pm_ext_flash)
  end_address: 0x6000
  region: external_flash
  size: 0x6000
mcuboot:
  address: 0x0
  end_address: 0xc000
  placement:
    before:
    - mcuboot_primary
  region: flash_primary
  size: 0xc000
mcuboot_pad:
  address: 0x10000
  end_address: 0x10200
  placement:
    align:
      start: 0x8000
    before:
    - mcuboot_primary_app
  region: flash_primary
  size: 0x200
mcuboot_primary:
  address: 0x10000
  end_address: 0x80000
  orig_span: &id001
  - app
  - tfm
  - mcuboot_pad
  region: flash_primary
  sharers: 0x1
  size: 0x70000
  span: *id001
mcuboot_primary_app:
  address: 0x10200
  end_address: 0x80000
  orig_span: &id002
  - app
  - tfm
  region: flash_primary
  size: 0x6fe00
  span: *id002
mcuboot_secondary:
  address: 0x80000
  end_address: 0xf0000
  placement:
    after:
    - mcuboot_primary
    align:
      start: 0x8000
  region: flash_primary
  share_size:
  - mcuboot_primary
  size: 0x70000
mcuboot_sram:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id003
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id003
nrf_modem_lib_ctrl:
  address: 0x20008000
  end_address: 0x200084e8
  inside:
  - sram_nonsecure
  placement:
    after:
    - tfm_sram
    - start
  region: sram_primary
  size: 0x4e8
nrf_modem_lib_rx:
  address: 0x2000a568
  end_address: 0x2000c568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_tx
  region: sram_primary
  size: 0x2000
nrf_modem_lib_sram:
  address: 0x20008000
  end_address: 0x2000c568
  orig_span: &id004
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x4568
  span: *id004
nrf_modem_lib_tx:
  address: 0x200084e8
  end_address: 0x2000a568
  inside:
  - sram_nonsecure
  placement:
    after:
    - nrf_modem_lib_ctrl
  region: sram_primary
  size: 0x2080
otp:
  address: 0xff8108
  end_address: 0xff83fc
  region: otp
  size: 0x2f4
sram_nonsecure:
  address: 0x20008000
  end_address: 0x20040000
  orig_span: &id005
  - sram_primary
  - nrf_modem_lib_ctrl
  - nrf_modem_lib_tx
  - nrf_modem_lib_rx
  region: sram_primary
  size: 0x38000
  span: *id005
sram_primary:
  address: 0x2000c568
  end_address: 0x20040000
  region: sram_primary
  size: 0x33a98
sram_secure:
  address: 0x20000000
  end_address: 0x20008000
  orig_span: &id006
  - tfm_sram
  region: sram_primary
  size: 0x8000
  span: *id006
tfm:
  address: 0x10200
  end_address: 0x18000
  inside:
  - mcuboot_primary_app
  placement:
    before:
    - app
  region: flash_primary
  size: 0x7e00
tfm_nonsecure:
  address: 0x18000
  end_address: 0x80000
  orig_span: &id007
  - app
  region: flash_primary
  size: 0x68000
  span: *id007
tfm_secure:
  address: 0x10000
  end_address: 0x18000
  orig_span: &id008
  - mcuboot_pad
  - tfm
  region: flash_primary
  size: 0x8000
  span: *id008
tfm_sram:
  address: 0x20000000
  end_address: 0x20008000
  inside:
  - sram_secure
  placement:
    after:
    - start
  region: sram_primary
  size: 0x8000

And this is the partition output:

  external_flash (0x800000 - 8192kB):
+--------------------------------------------+
| 0x0: littlefs_storage (0x6000 - 24kB)      |
| 0x6000: external_flash (0x7fa000 - 8168kB) |
+--------------------------------------------+

  flash_primary (0x100000 - 1024kB):
+--------------------------------------------------+
| 0x0: mcuboot (0xc000 - 48kB)                     |
| 0xc000: EMPTY_0 (0x4000 - 16kB)                  |
+---0x10000: mcuboot_primary (0x70000 - 448kB)-----+
+---0x10000: tfm_secure (0x8000 - 32kB)------------+
| 0x10000: mcuboot_pad (0x200 - 512B)              |
+---0x10200: mcuboot_primary_app (0x6fe00 - 447kB)-+
| 0x10200: tfm (0x7e00 - 31kB)                     |
+---0x18000: tfm_nonsecure (0x68000 - 416kB)-------+
| 0x18000: app (0x68000 - 416kB)                   |
+--------------------------------------------------+
| 0x80000: mcuboot_secondary (0x70000 - 448kB)     |
| 0xf0000: EMPTY_1 (0x10000 - 64kB)                |
+--------------------------------------------------+

  otp (0x2f4 - 756B):
+------------------------------+
| 0xff8108: otp (0x2f4 - 756B) |
+------------------------------+

  sram_primary (0x40000 - 256kB):
+--------------------------------------------------+
+---0x20000000: mcuboot_sram (0x8000 - 32kB)-------+
+---0x20000000: sram_secure (0x8000 - 32kB)--------+
| 0x20000000: tfm_sram (0x8000 - 32kB)             |
+---0x20008000: sram_nonsecure (0x38000 - 224kB)---+
+---0x20008000: nrf_modem_lib_sram (0x4568 - 17kB)-+
| 0x20008000: nrf_modem_lib_ctrl (0x4e8 - 1kB)     |
| 0x200084e8: nrf_modem_lib_tx (0x2080 - 8kB)      |
| 0x2000a568: nrf_modem_lib_rx (0x2000 - 8kB)      |
+--------------------------------------------------+
| 0x2000c568: sram_primary (0x33a98 - 206kB)       |
+--------------------------------------------------+

[Vidar Berg]

Hi,

Thank you for providing the partition report. It shows that the 'app' is placed in the tfm_nonsecure region, but mcuboot_secondary is not. This means the secondary slot will not be accessible to the application running in the non-secure processing environment. I'm unsure if mcuboot_secondary should be placed in the tfm_nonsecure region or in nonsecure_storage. I need to look into that. In the meantime, please try expanding the tfm_nonsecure region to include mcuboot_secondary as well.

I was wrong. The secondary slot should not be included in the tfm_nonsecure region, nor in nonsecure_storage. Please check if this code is greyed out in your project:

Security attribute is correctly set to non_secure when I try to reproduce it here: