Flash partitioning

[David Ormand]

The MCUboot, Slot0, and Slot1 flash partitions are defined in the board file. E.g., nrf52840dk_nrf52840.dts has boot_partition at 0 with length 0xC000, slot0_partition at 0xC000 with length 0x76000, slot1_partition at 0x82000 with length 0x76000, and storage_partition picking up at 0xF8000, right after Slot1.

In a Sigurd Hellesvik git repo, I learned about partition_manager_report; "west build -t partition_manager_report", which seems to say that Slot 0, for the primary image, is at 0xC000 with a length of 0x7A000 (starting with a 0x200 "pad") and Slot 1, for the secondary image, is at 0x86000 with a length of 0x7A000. It would seem the partition manager is free to allocate flash0 in nrf52840_qiaa.dtsi however it likes. Is the stuff in the DTS just a suggestion? This would seem to conflict with CONFIG_USE_DT_CODE_PARTITION=y in the prj.conf.

If I put stuff like USB in MCUboot and have to grow the partition to fit it by setting CONFIG_PM_PARTITION_SIZE_MCUBOOT, it reworks Slot0 and Slot1 lengths to fit in the 1Mbyte flash.

If I set CONFIG_NVS=y, partition_manager_report now has a nvs_storage block, not starting at 0xF8000 per the DTS file, but actually CONFIG_PM_PARTITION_SIZE_NVS_STORAGE from the end. So we really can't use the DTS macros to determine where a user-defined constant in flash might be.

The application I'm working on currently reserves some memory at the very end, to be loaded with a device serial number via nrfjprog. I'm "reserving" it in a DTS overlay and giving it a symbol that can be recovered with a DT macro. But if partition manager isn't really paying attention to the partition allocations in DT, how would I tell partition manager to reserve the space and not potentially overwrite it with a DFU or some NVS operation?

[Sigurd Hellesvik]

Hi,

Is the stuff in the DTS just a suggestion?

The Partition Manager will override DTS.

CONFIG_USE_DT_CODE_PARTITION

I think we "technically" in the build system use this even with the Partition Manager, we just do it a bit differently.

But if partition manager isn't really paying attention to the partition allocations in DT, how would I tell partition manager to reserve the space and not potentially overwrite it with a DFU or some NVS operation?

See this devacademy lesson for some background.

But to answer what I think you ask for:

DFU only updates mcuboot_primary and mcuboot_secondary, so the rest of your partitions should be unchanged.

Regards,
Sigurd Hellesvik

[David Ormand]

That's correct and i knew that; I shouldn't have mentioned DFU (even if that's my current bugbear).  For a Bluetooth device, for storing bond information in a non-volatile way, NVS is set up, and Partition Manager allocates a chunk of flash all the way to the end.  Maybe I've been lucky, and the bond info is stored at the top of this chunk of flash working down, and it's never gotten to the bit that I've reserved for the serial number.  But I suspect that with repeated bonding/unpairing, that region of flash used for bond info will creep down and eventually overwrite what I had naively thought I had reserved for the serial number.

[David Ormand]

I'm trying to anticipate or counter a potential problem.

This project I've got will use NVS as a data storing facility.  It's also a Bluetooth Central, so the bonding info will be saved.  I'm guessing the storage_partition is used for both purposes.  Maybe the bonding info is just NVS records.

I'm also expecting that the flash management method is to mark deleted records rather than actually erasing them, and acquire another flash page as the one currently in use is exhausted.  At some point the management system would search the pages for one consisting entirely of deleted records and erase that page.  But probably not until the end of the memory region was reached.  That is, the end of flash memory.

But if I have critical information written by an external tool like nrfjprog into a known memory location in flash at a point in product manufacture, and eventually NVS creeps up to and overwrites this information, then units are going to be failing in the field some (possibly lengthy) time after starting their service.

I was hoping that I could split off a little partition at the end of flash memory (I guess it would have to at least be a page) for this purpose in the Device Tree, but this won't work if Partition Manager doesn't respect it.

[David Ormand]

You know what, it just dawned on me there is a much better solution.  This is what UICR is intended for.  I should change the approach I implemented in the previous product software version.  It's still a fixed memory location suitable for coding into the nrfjprog script.

Although I think the question is still useful:  How can users assert control over how Partition Manager assigns flash memory?  Can a new partition be introduced in a way that Partition Manager would respect and manage it?

[Sigurd Hellesvik]

Short answer: UICR looks like what you need. Specifically CUSTOMEr.

This cannot be changed, and you have limited space.

NVS is more for changeable persistent data, or data that you will add regularly (aka large amounts in the end)

I will return later and write answers to your context questions.

Maybe Persistent storage of keys and data using the nRF Connect SDK would be of interest in the meantime

[David Ormand]

Thanks. That blog article is very helpful. I will have more questions about using flash when my project gets to that point.

I've been playing with UICR and it will do exactly what I want for semi-permanently storing the unit serial number, and access it from within the code as *(uint32_t *)(0x10001080). Super!

Now: I hope you can explain this partition management business to me.

Among other things, this project:
1. Uses BLE and saves bonding info, so I've got Settings and BT Settings enabled
2. Uses NVS for saving program data in flash, to survive power cycles
3. Will accept DFU via "serial recovery" in the bootloader.

The relevant config selections:
CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_FLASH_PAGE_LAYOUT=y
CONFIG_NVS=y
CONFIG_SETTINGS=y
CONFIG_BT_SETTINGS=y
CONFIG_BOOTLOADER_MCUBOOT=y

For NVS, I'm following the NVS sample. I'm allocating three 4K sectors

fs.sector_count = 3U;

The DTS for the nRF52840DK has storage_partition starting at 0xf8000. The code has

#define NVS_PARTITION storage_partition
#define NVS_PARTITION_DEVICE FIXED_PARTITION_DEVICE(NVS_PARTITION)
#define NVS_PARTITION_OFFSET FIXED_PARTITION_OFFSET(NVS_PARTITION)

If I print

fs.offset = NVS_PARTITION_OFFSET;

I get 0xfe000.

As previously explained, Partition Manager is overriding the Device Tree allocations. However, so happens this is 0x2000 before the end of memory. How is it allocating three sectors (without throwing any errors) when there's only two available?

And where would it be putting the settings?

If I look through build/zephyr/.config, I find

CONFIG_PM_PARTITION_SIZE_SETTINGS_STORAGE=0x2000

and if I get a partition_manager_report, I get

flash_primary (0x100000 - 1024kB):
+-------------------------------------------------+
| 0x0: mcuboot (0xc000 - 48kB) |
+---0xc000: mcuboot_primary (0x79000 - 484kB)-----+
| 0xc000: mcuboot_pad (0x200 - 512B) |
+---0xc200: mcuboot_primary_app (0x78e00 - 483kB)-+
| 0xc200: app (0x78e00 - 483kB) |
+-------------------------------------------------+
| 0x85000: mcuboot_secondary (0x79000 - 484kB) |
| 0xfe000: settings_storage (0x2000 - 8kB) |
+-------------------------------------------------+

What I believe I'm seeing is that there is no NVS partition reported, just a block for the settings, and there's two 4K sectors. And the space for the secondary image for DFU runs right up to the start of the settings block.

Apparently, I can't allocate space for an NVS partition separately from the NVS settings space. The symbol

CONFIG_PM_PARTITION_SIZE_NVS_STORAGE

depends on

CONFIG_NVS && !CONFIG_SETTINGS_NVS

and CONFIG_SETTINGS_NVS is True if CONFIG_SETTINGS and CONFIG_NVS are True.

[David Ormand]

Or perhaps I'm laboring under the misconception that Settings with the NVS backend is using a different flash area than NVS for program data storage.  Perhaps the bond info, and anything else stored as Settings, are mixed in with the entries that the program makes for other purposes.  But then it would seem that Settings would have an allocation of ID numbers separate from the set available for NVS filesystem.

Even if this is true, seems odd that nvs_mount() would allow setting up the filesystem in three sectors when there are two available.

[David Ormand]

Interestingly, if CONFIG_PM_PARTITION_SIZE_SETTINGS_STORAGE is left at the default 2 sectors, and NVS is set up at runtime with 3 sectors, nvs_mount() won't complain, but (at least) nvs_read() will fail with -EINVAL.  So the PARTITION_SIZE has to be set at least as large as what is requested for the NVS filesystem.  I do that, and nvs_read() doesn't complain anymore.

I have not yet got to the point of seeing how well NVS for data storage cooperates with NVS for BLE settings.

[David Ormand]

My application has a serial command interface (sort of "shell") in which a command writes a 20-byte object to NVS.  I've rigged this command to print what it is about to write.  I've rigged another command to dump a portion at the top of the Settings partition.  Which is sized at 0x3000 in prj.conf (per above).

Start: clear out all flash with nrfjprog --eraseall and then west flash the bootloader and application.

Note that the NVS filesystem is initialized before bt_enable() and settings_load() are called.

When I dump the partition portion before doing anything else, I get

000FD000 : FFFF8001 AC2807FA E8DA3D07 968B0056
000FD000 : A6C9D78D 682F7462 FF687361 FFFFFFFF
000FD000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FD000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF

I'm guessing this is something the BLE subsystem writes into the settings.

Then I command writing the following object to ID 1 in NVS:

73646401 0000007B FFFFFF00 12FFFFFF 00000034

and then dump the partition portion again.

000FD000 : 73640001 0000007A E8DA3D00 128B0056
000FD000 : 00000004 682F7462 FF687361 FFFFFFFF
000FD000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FD000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF

nvs_write() is overwriting the previous info.  Therefore I conclude that NVS filesystem for program data storage and NVS for Settings are not compatible.  I'm going to have to set up separate partitions.

[David Ormand]

The solution I've adopted is inspired by https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/scripts/partition_manager/partition_manager.html

1. Copy build/partitions.yml to pm_static.yml,

2. Edit to remove everything in each partition section except for address and size.

3. Delete the partitions for mcuboot_primary, mcuboot_primary_app, and mcuboot_pad.  These partition definitions will be supplied by Partition Manager from MCUboot's allocations.  An error in the build process informs me there must be exactly one gap in the allocation to be used for "app" (which seems to conflict with what the above page says about the default "flash_primary" region). 

4. Insert a new partition, in my case "nvs_storage".

5. By default, the Settings subsystem uses the "settings_storage" partition.  The default size of this partition is 2 sectors, 0x2000, or 8K.  The Zephyr authorities thought 2 would be enough.  I don't think I'm going to be using NVS for program data storage much more than BLE would be, so I'll set size for "nvs_storage" to 0x2000 also.

6. The only place to steal 2 sectors is from the primary and secondary image slots.  They should be the same.  Therefore, steal 1 sector from the gap to eventually be filled in by "mcuboot_primary" and 1 sector from "mcuboot_secondary".  Additionally, adjust the address for "mcuboot_secondary" up by one sector to be adjacent to "mcuboot_primary".

In this case, the resulting pm_static.yml looks like:

mcuboot:
  address: 0x0
  size: 0xc000
# Gap from 0x000C000 to 0x00084000 reserved for "app"
# This gap will be filled in with what MCUboot allocates,
# in this case, mcuboot_primary, mcuboot_primary_app, and mcuboot_pad.
mcuboot_secondary:
  address: 0x84000
  size: 0x78000
nvs_storage:
  address: 0xfc000
  size: 0x2000
settings_storage:
  address: 0xfe000
  size: 0x2000

7. In the project source file that implements the NVS filesystem, change the flash device designation to:

#define NVS_PARTITION           nvs_storage

and remember to load fs.sector_count with 2.

Now, when I build and flash the project and issue my command to dump memory (altered to report the tops of both the NVS and Settings partitions), I get

000FC000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FC000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FC000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FC000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FC000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FC000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FE000 : FFFF8001 AC2807FA E8DA3D07 968B0056
000FE000 : A6C9D78D 682F7462 FF687361 FFFFFFFF
000FE000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FE000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FE000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FE000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF

The two-sector Settings block at the end of memory starts at 0xFE000, and there is the same thing I saw earlier.  Whatever it is.

Then I issue some commands to write entries into the NVS filesystem.  Dumping the memory, I get

000FC000 : 73646401 0000007B FFFFFF00 12FFFFFF
000FC000 : 00000034 73646401 0000007C FFFFFF00
000FC000 : 12FFFFFF FFFFFF34 FFFFFFFF FFFFFFFF
000FC000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FC000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FC000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FE000 : FFFF8001 AC2807FA E8DA3D07 968B0056
000FE000 : A6C9D78D 682F7462 FF687361 FFFFFFFF
000FE000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FE000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FE000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
000FE000 : FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF

The written data is showing up in the hacked-in partition subsequently dedicated to the NVS filesystem.  NVS for data storage and NVS for BLE settings are no longer in conflict.

There may be a more elegant way to accomplish this than essentially forcing Partition Manager to accept manually-assigned allocations, and I would like to know if there is.

[Sigurd Hellesvik]

Hi again,

We hade some national holidays here, so things became a bit delayed.

I see that you found a solution, and if that works for you, no need to change it.
But I will try to explain it all from the bottom up, in case you are interested.

Zephyr vs nRF Connect SDK

Some years ago, we only had the nRF5 SDK. But to keep up with the times, we needed an RTOS. For this, we chose the Zephyr RTOS. However, we needed some leeway, so we created the nRF Connect SDK, which is basically the Zephyr RTOS with our own stuff on top.
Some info on this can be found in nRF Connect SDK and nRF5 SDK statement .

I will refer to only Zephyr (without the nRF Connect SDK) as Vanilla Zephyr.

Zephyr DTS and Partition Manager

Zephyr use DeviceTree (DTS) to configure the hardware. Among this DTS config there are partition information. However, for the nRF Connect SDK we needed something more dynamic than DTS partitioning. So for the nRF Connect SDK we created the Partition Manager.
This means that the Partition Manager is only in the nRF Connect SDK and not in Vanilla Zephyr.
And the nRF Connect SDK uses Zephyr but not vice versa.
So you can use DTS partitioning in the nRF Connect SDK but not the Partition Manager in Vanilla Zephyr.

This means that in the nRF Connect SDK, you have two alternatives for partitioning: DTS partitioning or Partition Manager.
To complicate things a bit: The build system still uses DTS partitioning, so the partition manager technically generates DTS partitioning I think.
But from a user perspective, you have must use only one to configure your project.
=> You can not configure partitions with DTS partitioning and the Partition Manager at the same time. If you do, DTS partitioning will be ignored and the Parititon Manager will "win".

How can you know if your project uses DTS partitioning or the Partition Manager?
If the Parititon Manager is enabled, that is used. If not, DTS partitioning is used. You can check CONFIG_PARTITION_MANAGER_ENABLED to see which one is used.

How to work with static partitioning

Okay, so the whole point of the Partition Manager is that it is dynamic.
Then why even allow for static partitioning?
The reason is twofold:

1. MCUboot requires that partitioning remain the same between DFUs
2. To create custom partitions

While you technically can use pm_static.yml in your project at any time, i would argue that you should only do it for one or both of the above reasons. The reason for this is mostly that working with pm_static.yml is hard, because we need to apply our feeble human brains to align hex values manually, instead of making the build system do this automatically for you.

How to do static partitioning for MCUboot

1. Backup your whole build folder from your release build. This is generally a good idea.
2. Copy build/partitions.yml to pm_static.yml for your project. This will make sure that project partitioning never changes as long as pm_static.yml stays the same.

As you can see, this does not require you to edit pm_static.yml manually.

How to create a custom partition

1. Create a pm_static.yml file
2. Add only your custom partition
3. The partition manager should now dynamically add partitions around your custom partition

Here, you can see that you only need to add custom partitions in pm_static.yml, so you do need minimal editing of pm_static.yml

Where are Partition Manager partitions defined

These are defined in one of two places:

1. ncs/nrf/subsys/partition_manager, see pm.yml files
2. Child image samples with pm.yml files.
   1. Examples:
      1. mcuboot
      2. netboot (b0n)
      3. NSIB (b0)

You should not change these, but they are good to know about for debugging sometimes.

NVS, Settings, and general file systems

So NVS, Settings and file systems are kinda corner cases.

Firstly, if you enable for example NVS in your project, the partition manager will automatically add a partition named nvs_storage.

This means that our internal uses of NVS will use nvs_storage automatically.
You can still use nvs_storage from your application. However, if you use NVS in your application, you can also point your app's instance of NVS to another partition, usually a custom NVS partition. The reason to use a custom partition is related to flash_map_pm.h, explained below.

Then we also have nrf/include/flash_map_pm.h you can see that we define a partition named "storage_partitions" for multiple different file sysstems. Because of this, if you add both NVS and littlefs to a project, these may end up using the same partition. I must admit that I am a bit unsure to how this will work in practice.

[David Ormand]

Thank you for this excellent explanation and the background info.  Since I am approaching Zephyr from a Nordic point-of-view, I'm aware there are differences between Zephyr as part of NCS and "vanilla" Zephyr from https://www.zephyrproject.org/ - which states that a number of platforms are supported, including Nordic, but not any notion of how far a "fork" for that support may go.